Submit Spark in Java : No client available issue

Describe the bug
Steps to reproduce

Package Spark JAR in java
sparkApplication.startApplication()
No client available

test result

Spark JAR in scala is working fine.
cc.blynk.clickhouse.ClickHouseDriver is working fine.
Run in DIRECT Code is working fine. - follow code
only not working com.clickhouse.jdbc.ClickHouseDriver and spark submit in JAVA Jar

public class StartTest {
    @Test
    public void testStart() throws Exception {
        // String path = "/Users/sofriend/storage/job/2001.json";
        // Start.main(new String[]{path});

        Properties prop = new Properties();
        // prop.setProperty("driver", "com.clickhouse.jdbc.ClickHouseDriver");
        prop.setProperty("user", "pduser");
        prop.setProperty("password", "rhtmxm");
        String url = "jdbc:clickhouse://<IP_ADDRESS>:9000/test";
        String tableName = "source_1679899992461000093";
        try {
            // Class.forName("com.clickhouse.jdbc.ClickHouseDriver");
            String countQuery = "(select count(1) from " + tableName + ") as pz";
            Row row = SparkUtil.getSparkSession().read().jdbc(url, countQuery, prop).first();
            System.out.println( row.getDecimal(0));
            Row row2 = SparkUtil.getSparkSession().read().jdbc(url, countQuery, prop).first();
            System.out.println( row2.getDecimal(0));
        }
        catch(Exception e) {
            e.printStackTrace();
        }
    }
}

only not working in spark submit java.

ClickHouseDriver: ClickHouse Driver <IP_ADDRESS>(JDBC: <IP_ADDRESS>) registered

is ok? <IP_ADDRESS>

Handling java.net.ConnectException: No client available(failover=0, retry=0)
this is problem.

follow log
[14:35:07.339] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 **DEBUG ClickHouseDriver: ClickHouse Driver <IP_ADDRESS>(JDBC: <IP_ADDRESS>) registered**
[14:35:07.341] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG JDBCOptions: Keytab path found, assuming manual upload
[14:35:07.352] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ConnectionProvider: Loaded built-in provider: org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider@11399548
[14:35:07.352] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ConnectionProvider: Loaded built-in provider: org.apache.spark.sql.execution.datasources.jdbc.connection.DB2ConnectionProvider@58164e9a
[14:35:07.352] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ConnectionProvider: Loaded built-in provider: org.apache.spark.sql.execution.datasources.jdbc.connection.MariaDBConnectionProvider@76e6c070
[14:35:07.352] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ConnectionProvider: Loaded built-in provider: org.apache.spark.sql.execution.datasources.jdbc.connection.MSSQLConnectionProvider@f03ee8f
[14:35:07.352] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ConnectionProvider: Loaded built-in provider: org.apache.spark.sql.execution.datasources.jdbc.connection.PostgresConnectionProvider@5e5a8718
[14:35:07.353] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ConnectionProvider: Loaded built-in provider: org.apache.spark.sql.execution.datasources.jdbc.connection.OracleConnectionProvider@4e50ae56
5
[14:35:07.353] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG JDBCOptions: Keytab path found, assuming manual upload
2
[14:35:07.354] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG JDBCOptions: Keytab path found, assuming manual upload
[14:35:07.354] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG BasicConnectionPro**vider: JDBC connection initiated with URL: jdbc:clickhouse://<IP_ADDRESS>:8123/test and properties: {password=rhtmxm,** driver=com.clickhouse.jdbc.ClickHouseDriver, dbtable=(select count(1) from source_1679903931850000201) as pz, user=pduser, url=jdbc:clickhouse://<IP_ADDRESS>:8123/test}
[14:35:07.354] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ClickHouseDriver: Creating connection
[14:35:07.376] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ClickHouseConnectionImpl: Connecting to: ClickHouseNode [uri=http://<IP_ADDRESS>:8123/test, options={driver=com.clickhouse.jdbc.ClickHouseDriver,dbtable=(select count(1) from source_1679903931850000201) as pz,url=jdbc:clickhouse://<IP_ADDRESS>:8123/test}]@1398196566
[14:35:07.376] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 DEBUG ClickHouseClientBuilder$Agent: Failed to find client for ClickHouseNode [uri=http://<IP_ADDRESS>:8123/test, options={driver=com.clickhouse.jdbc.ClickHouseDriver,dbtable=(select count(1) from source_1679903931850000201) as pz,url=jdbc:clickhouse://<IP_ADDRESS>:8123/test}]@1398196566
[14:35:07.376] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 23/03/28 14:35:07 **DEBUG ClickHouseClientBuilder$Agent: Handling java.net.ConnectException: No client available(failover=0, retry=0)
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] java.sql.SQLException: No client available, server ClickHouseNode [uri=http://<IP_ADDRESS>:8123/test, options={driver=com.clickhouse.jdbc.ClickHouseDriver,dbtable=(select count(1) from source_1679903931850000201) as pz,url=jdbc:clickhouse://<IP_ADDRESS>:8123/test}]@1398196566**
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:85)
SqlExceptionUtils.java:85
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.SqlExceptionUtils.create(SqlExceptionUtils.java:31)
SqlExceptionUtils.java:31
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.SqlExceptionUtils.handle(SqlExceptionUtils.java:90)
SqlExceptionUtils.java:90
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.getServerInfo(ClickHouseConnectionImpl.java:131)
ClickHouseConnectionImpl.java:131
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:334)
ClickHouseConnectionImpl.java:334
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:287)
ClickHouseConnectionImpl.java:287
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:157)
ClickHouseDriver.java:157
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at com.clickhouse.jdbc.ClickHouseDriver.connect(ClickHouseDriver.java:41)
ClickHouseDriver.java:41
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:49)
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at org.apache.spark.sql.execution.datasources.jdbc.connection.ConnectionProvider$.create(ConnectionProvider.scala:77)
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$createConnectionFactory$1(JdbcUtils.scala:64)
[14:35:07.377] [INFO ] org.apache.spark.launcher.app.Start      [] [:] 	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.getQueryOutputSchema(JDBCRDD.scala:62)
------------------

java 11
Clickhouse in docker

Hi @sofriendcom, judging from ClickHouse Driver <IP_ADDRESS>(JDBC: <IP_ADDRESS>), it looks like you're using a private build of JDBC driver? Please use latest version(v0.4.2) and add classifier(http or all) if the default artifact does not work for you.

@zhicwu  i use 0.4.2:all  and 0.4.1, 0.3.x.. all same issue..

Could you check classpath to understand where the ClickHouse Driver <IP_ADDRESS>(JDBC: <IP_ADDRESS>) came from? You probably have duplicated classes there, try search com.clickhouse in classpath including all jars.

this is work outside spark.  only not work in apache spark submit.

there is only one clickhouse driver.
this is my gradle dependency
dependencies {
    testImplementation group: 'junit', name: 'junit', version: '4.12'
    implementation ( group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.2.3')
    implementation ( group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.2.3')
    implementation 'org.codehaus.janino:janino:3.0.16'
    implementation group: 'org.postgresql', name: 'postgresql', version: '42.3.1'
    implementation "com.google.code.gson:gson:2.9.0"
    compileOnly 'org.projectlombok:lombok:1.18.24'
	annotationProcessor 'org.projectlombok:lombok:1.18.24'
    implementation ('com.clickhouse:clickhouse-jdbc:0.4.2:http')
    // implementation('cc.blynk.clickhouse:clickhouse4j:1.4.4')
}

and i'am packaging fat jar for spark submit.
task jar_spark(type: Jar) {
    baseName = 'patransform'
    // manifest {
    //     attributes 'Main-Class': 'com.puzzledata.transform.Start'
    // }
    zip64 true
    exclude("**/logback.xml", "**/license", "META-INF/**")
    duplicatesStrategy = DuplicatesStrategy.EXCLUDE
    from { configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) } }
    with jar
}

there is only two jdbc driver. postgresql and clickhouse.
I think that is bug why "cc.blynk.clickhouse:clickhouse4j:1.4.4" driver is working fine.

@sofriendcom did you managed to fix the issue?

It hasn't been fixed yet.
To solve this I am still using the "cc.blynk.clickhouse:clickhouse4j:1.4.4" driver.

Good day, @vitalvi!
We will look into the issue.
Btw, we have own implementation of Spark connector https://clickhouse.com/docs/en/integrations/apache-spark#spark-jdbc
May be it can solve the issue?
Thanks!


Fix SSL library usage is not thread safe

Openwsman randomly crashes with BT like:
#0  0x00007f95523a45f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f95523a5ce8 in __GI_abort () at abort.c:90
#2  0x00007f954decc208 in rb_bug (fmt=fmt@entry=0x7f954e012ebf "Segmentation fault") at error.c:314
#3  0x00007f954df795b6 in sigsegv (sig=, info=, ctx=) at signal.c:679
#4
#5  freelist_extract (for_read=for_read@entry=1, sz=sz@entry=17736, ctx=, ctx=) at s3_both.c:701
#6  0x00007f955316c373 in ssl3_setup_read_buffer (s=s@entry=0x7f9555886df0) at s3_both.c:770
#7  0x00007f955316c4d9 in ssl3_setup_buffers (s=0x7f9555886df0) at s3_both.c:827
#8  0x00007f955316da7d in ssl23_get_client_hello (s=0x7f9555886df0) at s23_srvr.c:266
#9  0x00007f955316e108 in ssl23_accept (s=0x7f9555886df0) at s23_srvr.c:210
#10 0x00007f95541f11ce in ssl_handshake (stream=stream@entry=0x7f95558dd4b8) at /usr/src/debug/openwsman-2.3.6/src/server/shttpd/io_ssl.c:39
#11 0x00007f95541ed0e6 in shttpd_add_socket (ctx=0x7f9555884530, sock=sock@entry=14, is_ssl=is_ssl@entry=1) at /usr/src/debug/openwsman-2.3.6/src/server/shttpd/shttpd.c:768
#12 0x00007f95541f443b in wsmand_start_server (ini=ini@entry=0x7f95556f0030) at /usr/src/debug/openwsman-2.3.6/src/server/wsmand-listener.c:750
#13 0x00007f95541eb866 in main (argc=, argv=) at /usr/src/debug/openwsman-2.3.6/src/server/wsmand.c:295
This patch makes usage of openssl thread safe and avoids this kind of crashes. It's based on examples from openssl upstream tarball:
crypto/threads/mttest.c
Also see:
https://www.openssl.org/docs/manmaster/crypto/threads.html

Thanks Vitezslav !
I still need to wrap my head around thread handling in wsman. There's wsman_setup_thread and max_threads, your code uses thread_setup and CRYPTO_num_locks (it seems). This is confusing.

Yeah, thread handling seems to be quite broken in wsman (or more precisely in shttpd, I would say). I can reproduce various thread issues (SSL thread safeness  is just one of them) by running several (e.g. six) wsman enumerate commands in a loop, wsman goes down in few hours at the best. The patch above helps to increase the living time:) Feel free to rename functions/variables names as it suits you, or let me know them and I will do that.
By the way, maybe the locking isn't done correctly, because I see regularly failing assert in src/server/shttpd/shttpd.c:1017 (count of connections bellow zero).

shttpd is an external component, copied (more or less) 1:1 into Openwsman years ago. It might be feasible to replace it with a newer version of shttpd (if it exists).

shttpd is at http://docs.huihoo.com/shttpd/. Latest version is 1.42 (from 2008).

I see... The API and internals had changed a bit in shttpd 1.42, but I managed to make it work somehow. It still needs to be polished and some testing needs to be done - I really wonder if it will help to avoid those SSL/race conditions problems. I'll let you know.
Just FYI - it seems that shttpd was transformed into mongoose:
https://code.google.com/archive/p/mongoose/

I'll merge this now while looking closer to mongoose.


Letsencrypt pem generation failed

Hi!
I've just installed Roxy-wi, so far so good.
After I got used to it a bit, I wanted to setup valid SSL certificates. (for HAProxy)
I've done the frontend/backend setup, the key and cert is generated and signed, but the ansible wants to make the combined PEM file, and that is where I got an error:
['', 'PLAY [<IP_ADDRESS>] ***********************************************************', '', 'TASK [Gathering Facts] *********************************************************', 'ok: [<IP_ADDRESS>]', '', 'TASK [Install certbot] *********************************************************', 'ok: [<IP_ADDRESS>]', '', 'TASK [Kill cerbot standalone] **************************************************', 'changed: [<IP_ADDRESS>]', '', 'TASK [Get cert] ****************************************************************', 'changed: [<IP_ADDRESS>]', '', 'TASK [Combine into pem file] ***************************************************', 'fatal: [<IP_ADDRESS>]: FAILED! => {"changed": true, "cmd": "cat /etc/letsencrypt/live/proxy1.idonttellyou.hu/fullchain.pem /etc/letsencrypt/live/proxy1.idonttellyou.hu/privkey.pem > \\"error:\\"/\\"proxy1.idonttellyou.hu\\".pem", "delta": "0:00:00.002538", "end": "2023-06-13 14:54:53.393817", "msg": "non-zero return code", "rc": 2, "start": "2023-06-13 14:54:53.391279", "stderr": "/bin/sh: 1: cannot create error:/proxy1.idonttellyou.hu.pem: Directory nonexistent", "stderr_lines": ["/bin/sh: 1: cannot create error:/proxy1.idonttellyou.hu.pem: Directory nonexistent"], "stdout": "", "stdout_lines": []}', '', 'PLAY RECAP *********************************************************************', '<IP_ADDRESS> : ok=4 changed=2 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 ', '', "error: Can't create SSL certificate"]
The syslog tells me this:
Jun 13 14:54:53 Titan python3[20952]: ansible-ansible.legacy.command Invoked with _raw_params=cat /etc/letsencrypt/live/proxy1.idonttellyou.hu/fullchain.pem /etc/letsencrypt/live/proxy1.idonttellyou.hu/privkey.pem > "error:"/"proxy1.idonttellyou.hu".pem _uses_shell=True warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Thanks,
Peter

Hi,
Try to create the dir manually and run again

I would like to, but I don't know where should I do this.
If I'm right, this one is trying to make something into the / directly.
Where is the value of the {{SSL_PATH}} variable which used by ansible?
If you look at the command I think the SSL_PATH is empty
www-data   24289  0.2  0.0   4736  3500 ?        S    15:40   0:00 /bin/bash ./letsencrypt.sh PROXY= haproxy_dir=/etc/haproxy DOMAIN=proxy1.idonttellyou.hu<EMAIL_ADDRESS>SSH_PORT=22 SSL_PATH=error: The path must contain the name of the service. Check it in Roxy-WI settings HOST=<IP_ADDRESS> USER=username PASS= KEY=/var/lib/roxy-wi/keys/RoxyWi-key_Default.pem
I changed the ansible role to this, and now the pem is created
    - name: Combine into pem file
#      shell: cat /etc/letsencrypt/live/{{DOMAIN}}/fullchain.pem /etc/letsencrypt/live/{{DOMAIN}}/privkey.pem > "{{SSL_PATH}}"/"{{DOMAIN}}".pem
      shell: cat /etc/letsencrypt/live/{{DOMAIN}}/fullchain.pem /etc/letsencrypt/live/{{DOMAIN}}/privkey.pem > /etc/ssl/certs/{{DOMAIN}}.pem

However, the role cannot finish, because of the cronjob cannot be created:
...  'TASK [Creates cron jobs] *******************************************************', 'fatal: [<IP_ADDRESS>]: FAILED! => {"changed": false, "msg": "Failed to find required executable crontab in paths: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"}', '', 'PLAY RECAP *********************************************************************', '<IP_ADDRESS> : ok=7 changed=3 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 ', '', "error: Can't create SSL certificate"]
JFTR: the letsencrypt renew script, which should use the {{SSL_PATH}} also got an error when the template copied over:
`
[root(Titan:2)]  cat /etc/haproxy/scripts/renew_letsencrypt.sh
#!/bin/bash
cd /etc/letsencrypt/live/
<EMAIL_ADDRESS>path='error:'
command='certbot certonly --standalone'
`
BTW, I'm using Ubuntu 22 LTS Server.

Just a quick followup. My cron problem is caused by me, since I used minimal Ubuntu as base, and they not install cron...
About the SSL_PATH variable... I've done some research in the code, and I found the app/modules/config/add.py, where this variable got a value based on an SQL field:
ssl_path = common.return_nice_path(sql.get_setting('cert_path'))
The next line is haproxy_dir, which use a same method, but without this common.return_nice_path call, which looks like this:
def return_nice_path(return_path: str) -> str:
        if not check_is_service_folder(return_path):
                return 'error: The path must contain the name of the service. Check it in Roxy-WI settings'

        if return_path[-1] != '/':
                return_path += '/'

        return return_path

So somehow this check_is_service_folder function returns as False, but I don't understand the logic behind it.
After I changed the ssl_path to give a value without a nice path ssl_path = sql.get_setting('cert_path')  I was able to request the LE certificate.
Hope this helps to point out something.
If it matters, my cert_path is configured to /etc/ssl/certs/ and I have manual installation.

It looks like a bug, will fix it in the next release, thank you

Fixed https://github.com/hap-wi/roxy-wi/commit/fb3b1d186789eb65ad53623cc1517c2d776bbdaa


Update_with_password deletes associated objects

Rails: 4.1.4  Devise: 3.4.0  Warden: 1.2.3
Hello.
I have a  model User which is similiar to:
class User < ActiveRecord::Base
has_one :profile,     inverse_of: :user, dependent: :destroy
accepts_nested_attributes_for :profile
validate :profile, presence: true
devise :database_authenticatable, :registerable, :recoverable, :rememberable, :validatable
end
The edit form updates the user and the profile at the same time. I noticed that when the update fails (blank current password or invalid profile inputs) and I leave the edit page, the profile is deleted. I found that the problem is from the method update_with_password (update_without_password works fine). After removing these two lines everything works Ok again:
self.assign_attributes(params, *options)
self.valid?

Closing as it looks like a bug in Rails.


Make identifier argument of create_snapshot optional

As suggested in https://github.com/westonganger/active_snapshot/issues/13#issuecomment-965690730 , we could make the identifier option of create_snapshot optional. If someone does not set it then we could utilize the id of the record
class SnapshotItem
  validates :identifier, presence: true, unique: {...}, allow_nil: true

  def identifier
    self[:identifier] || id
  end
end

Not sure how I feel about automatically setting some identifier value that are not conciously thought about by the developer. So would like some feedback on this.

I dont think we need to set the identifier at all. If anyone needs to find a snapshot they can use the created_at timestamp column.

Merged to master on Sept 19, will update this issue once a new version is released.


searching an oasis vs searching central

Differences in searching an oasis where the sample id, project_id, experiment_id are more relevant as compared to on the central.

We have a more flexible search with specialised and customisable search apps.


AbstractSimiWidget.renderButton is incompatible with yarn mapping

Describe the Suggestion
AbstractSimiWidget.renderButton has the same method signature with Widget.renderButton (mojmap: AbstractWidget.renderWidget), causing all projects which use yarn mapping to fail to load.
Screenshots and Videos
No response
Additional Context
No response

wait, wrong branch, let me see if it's changed

okay, that has changed. It should probably be renamed on forge too.

fix PR opened: https://github.com/Creators-of-Create/Create/pull/5558


Reconnecting WebSockets

Sets up event watcher for websockets end or error.
TimeNode will attempt to reconnect when event for websockets end or error is received.
Config takes a new optional endpoints and maxRetries parameter which will be the endpoints that are cycled through as the TimeNode attempts to reconnect or how many times it attempts reconnects.

Pull Request Test Coverage Report for Build 1760

1 of 12 (8.33%)  changed or added relevant lines in 1 file are covered.
15 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.4%) to 79.202%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/TimeNode.ts
1
12
8.33%

Files with Coverage Reduction
New Missed Lines
%

src/Actions/Pending.ts
2
88.64%

src/EconomicStrategy/EconomicStrategyHelpers.ts
13
82.64%

Totals

Change from base Build 1756:
0.4%

Covered Lines:
764

Relevant Lines:
932

💛  - Coveralls

Haven't tested that but this seems to restart scanning, while we want to reconnect to wss (like creating new provider and connect).
Just make sure it works, there are some good tools like Chrome Network tab where you can simulate disconnections.

This also may end up with infinite loops when starting will fail.
I think the best way to go would be to fork web3-ws-provider and replace the websocket provider used in it with something like reconnecting-websocket that have more sophisticated reconnection strategies.

@kosecki123 :ok_hand: Will make a WsReconnect class.


fix pip install in drone script

With this PR, we fix all warnings we could have in CI.
The only warning we are explicitly ignoring after that is unused-function. Since it's a problem with GCC6-8 that's already been fixed in GCC, there's nothing we can do about it. At most, we could include a link to the bug.
In the end, all we needed to fix the script was to upgrade pip. The cmake subdir test was also simplified to make errors there much less likely.

GHA caught an error that Drone didn't catch. I'll fix it.


Add JSON foreign data wrapper

What feature are you requesting?
In addition to CSV and Parquet files, we should be able to read JSON.
Why are you requesting this feature?
Support for JSON format.
What is your proposed implementation for this feature?
The JSON FDW can be created exactly like the CSV/Parquet wrappers, as DuckDB has an API for reading JSON.
Full Name:
Ming
Affiliation:
ParadeDB

This is a particularly good first issue as we have existing users requesting JSON support

I'll take this up!

I'll take this up!

Thank you 🙏


fixes #356

Overview
when status is error, still generates final output
Related Issue / Discussion
#356
Additional Information
Contribution Agreement
(as per https://github.com/geopython/pywps/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

[ ] I'd like to contribute [feature X|bugfix Y|docs|something else] to PyWPS. I confirm that my contributions to PyWPS will be compatible with the PyWPS license guidelines at the time of contribution.
[x] I have already previously agreed to the PyWPS Contributions and Licensing Guidelines

Coverage decreased (-0.03%) to 73.101% when pulling 06f06d334b4ec3b4529836b80921660cf643ff05 on jachym:356_fix_exception_output into 71d16b97cc750844ced3fe2009b87cb28b18e8a9 on geopython:master.


CI: Update checkout settings

Change-type: patch

@Page- apparently I'm not a reviewer here because I don't have write access

x

LGTM


[SPARK-43316][SQL] Add more CTE SQL tests

What changes were proposed in this pull request?
This PR borrows CTE SQL test coverage from other databases (Postgres, ZetaSQL, DuckDB).

Why are the changes needed?
CTE is a hot area in terms of regression and needs more test coverage.

Does this PR introduce any user-facing change?
No user-facing change.

How was this patch tested?
This PR adds SQL tests only.

seems scala style check failed

@gengliangwang could you help merge it? Thanks

thanks, merging to master!


Usage of Package Microsoft.Azure.Functions.Worker.Sdk in provided sample Function App

If you try to build the FunctionApp sample using following docker image mcr.microsoft.com/dotnet/sdk:5.0, it will fail.
docker pull mcr.microsoft.com/dotnet/sdk:5.0
docker run --rm -v $(pwd):/app -w /app/samples/FunctionApp mcr.microsoft.com/dotnet/sdk:5.0 dotnet publish -c release -o out
If you ommit these Package References  and  the build succeeds.
If you use Visual Studio 2019, net5 and net core 3.1 are installed so the build failure is not a problem. But the problem arose when deploying a static web app with a net 5 api. Oryx only installs net5 (I know net5 is not supported yet for the api)
My question is, why do we actually need the sdk?

I noticed on my Azure DevOps hosted agent (ubuntu-latest) it does not generate the .azurefunctions folder with the dotnet publish command. Switched to windows-latest and it generates the .azurefunctions folder as expected. Using v1.4.0 of the SDK.


Super fistbump

A :super_fistbump: emoji that user's can pass to others by using it in a recognition. This would be passed to the receiver and passed around from recognition to recognition. This would be worth 25 :fistbumps:

@someone :super_fistbump: here's your super cool award for doing that awesome thing!

Closed by #178


Add a Cypress test to open a listing page (and remove unused test files)

Issue

Closes #480

Description
This change adds a test that verifies that we can open a listing detail page without error. It also removes a few unused test files and refactors the package.json files in sites/public and at the top level.
Note: I tried to make it so that this test created its own listing as a "before" step. It took too long to figure out the permissions, so I gave up on that in favor of just assuming that there already exists at least one listing already in the database. This is a fair assumption, because the yarn interface defined in sites/public/package.json makes it so that the database is reseeded (with the seed listings) before the test is run.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Prototype/POC (not to merge)
[X] This change is a refactor/address technical debt
[ ] This change requires a documentation update
[ ] This change requires a SQL Script
[X] Test addition

How Can This Be Tested/Reviewed?
$ yarn test:app:public:headless
Checklist:

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[ ] I have reviewed the changes in a desktop view
[ ] I have reviewed the changes in a mobile view
[X] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[X] My changes generate no new warnings
[X] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ ] I have assigned reviewers
[ ] I have updated the changelog to include a description of my changes
[ ] I have run yarn generate:client if I made backend changes

Sorry @willrlin I'm looking into the test failure and have made this a draft for now (not ready to review). I'll convert it to a real PR and add a note here when it's ready for review.

@willrlin Okay this is ready for review now. I tried to get this working in a better way (with the test creating the listing that it relies on, instead of relying on db:reseed to create a listing), but I've sunk more time into this than I should've. I still think this is valuable (now we have a test that actually opens the individual listing page), but I don't think it warrants more time.


Missing use-before-def error for redeclared property present in base class

Bug Report

🔎 Search Terms

used before extends
property initialization base
useDefineForClassFields initialization

🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about useDefineForClassFields

⏯ Playground Link

Playground link with relevant code
💻 Code

// @useDeclareForClassFields: true
class Base {
    a = 1;
    b = 1;
    c = 1;
    d = 1;
    e = 1;
}

class Derived extends Base {
    a = 2;
    b;
    declare c: number;
    d: number = this.a /*2*/ + this.b /*undefined*/ + this.c /*1*/ + this.d /*1*/ + this.e /*1*/; // expected error on 'b', all other are valid
    e;

    constructor() {
        super();
        this.b = 2;
        this.e = 2;
    }
}

🙁 Actual behavior
No error in the code above. Use-before-def allows access to properties of the base class, even if the current class redeclares that property.
This is correct until you enable useDefineForClassFields. Then the redeclaration of b overrides the base class property before the access. I added comments containing the runtime values.
🙂 Expected behavior

a is allowed because it's initialized
b is a use-before-def error
c is allowed because of the declare modifier
d is allowed to access the base class property in the initializer becaues at that point its value is not overridden yet
e is allowed because its value is overridden after accessing it.

/cc @sandersn

There's a very similar issue when a base class property is redeclared as parameter property. Note that this one requires target: ESNext AND useDefineForClassFields:
class Base {
    a = 1;
}

class Derived extends Base {
    b = this.a /*undefined*/;

    constructor(public a: number) {
        super();
    }
}

For this case the error logic needs to be chaned to: if emitting native class fields, it's not allowed to use parameter properties in other property's initializers, regardless of the presence of a base class property with the same name.

There's a very similar issue when a base class property is redeclared as parameter property. Note that this one requires target: ESNext AND useDefineForClassFields:
class Base {
    a = 1;
}

class Derived extends Base {
    b = this.a /*undefined*/;

    constructor(public a: number) {
        super();
    }
}

For this case the error logic needs to be chaned to: if emitting native class fields, it's not allowed to use parameter properties in other property's initializers, regardless of the presence of a base class property with the same name.


SDK add Quota command and models. Add Tests for Quota

SDK pickup Quota files for  models and commands from - https://github.com/AzureSDKAutomation/azure-sdk-for-net/commits/sdkAutomation/Microsoft.Azure.Management.Reservations - Auto generated by SDK Team.
Add Tests for Quota.

We are in public preview. Please expedite the SDK release.
As we need  to the SDK public link at - https://docs.microsoft.com/en-us/rest/api/reserved-vm-instances/quotaapi

@erich-wang - Please review it. We need to release the SDK to public site, as the RP is in Public Preview.
Thanks.

@erich-wang _ Please review the PR. We need to publish the SDK to public site as the RP is in Public Preview.


[New feature] Manage command configuration

Add a new type of command launcher managed resource: "config". Each command can request to such configurations in  requestedResources property.
A package should also declare the list of configurations it requires so that the command launcher can provide a proper interface to access them.
Example manifest:
pkgName: demo-pkg
version: 1.0.0
cmds:
  - name: test-cmd
     type: executable
     requiredResources: [ 'CONFIG_NAME_1', 'CONFIG_NAME_2' ]

configs:
  - name: NAME_1
    type: string
    default: hello
  - name: NAME_2
     type: integer
     default: 10

Would this mean the values can be set up using the config command and that the storage would be managed by the launcher?
If so, and if the interface is env vars, then I think there would be some checks for clashes or some implicit namespacing.


added workaround to avoid empty k3s configs in screenshot

Added workaround to preserve k3s configuration files from deletion with k3s-uninstall

@bulnv Are you sure about this, did this change fix it for you?
Because write_files in cloud-init will create the required path if missing. But maybe indeed, the k3s uninstallation will delete it... If that's the case, it would explain it.

Then it's a matter of timing depending on the node. Great catch!! Will merge and do final testing later on.

Was a few days off, so I just read now this thread - but can't see really what this patch changes? What do you mean with timing? There could indeed be an issue dependent on when the snapshot has been created... unfortunately the current snapshot generation is unreliable - this is not the case anymore when the snapshot is created from the outside, e.g. via packer.

@ifeulner No worries. Now it's realiable, I just had changed the location of the k3s config file (you had it correct in the first place, tmp was the right place), and it got deleted by the k3s uninstall on some nodes depending on the node size I guess, so timing. This PR is correct and restores the bug that I introduced during another fix.


Shape visual issue

Information

Haze version: 0.5.3
Platform: Android13
Compose version: 1.6.0

Expected Behavior
Correct rounded corners without white lines.
Actual Behavior
The rounded corners have white lines.

Steps to Reproduce the Problem
Box {
  Image(
      modifier = Modifier
          .fillMaxSize()
          .haze(
              state = hazeState
          ),
      painter = painterResource(
          id = R.drawable.test
      ),
      contentDescription = "",
      contentScale = ContentScale.Crop
  )

  Box(
      modifier = Modifier
          .align(Alignment.Center)
          .padding(horizontal = 14.dp)
          .height(80.dp)
          .fillMaxWidth()
          .hazeChild(
              state = hazeState,
              shape = RoundedCornerShape(20.dp),
              style = HazeStyle(noiseFactor = 0f, tint = Color.Black.copy(.6f))
          ),
  )
}

So this is caused by rounding + anti-aliasing on our clip paths. We clip twice, once for the blurred shape, and clip out'ing the content behind. The antialiasing means that the edges are aliases and have some translucency. Since we're clipping twice, the float rounding means that there's a chance that the blurred content won't cover the very edges of the clipped out content behind (which is the white parts you're seeing).
I've got a hacky fix in #133 which basically shrinks the clip out shape by 1px on all sides, so that there's always something drawn behind the blurred content. This won't work for all shapes though, as we need to resize the outline. I'm going to have a think today if there's a better way.


Issue with textarea content

Let me first say thanks @davidodenwald for this plugin. The speed we're seeing over crappy djlint's formatting is astounding.
I wanted to raise an issue with textarea elements that I'm seeing:

        Description of data

    <textarea
        id="description"
        name="description"
        placeholder="Describe the data in the file..."
        class="textarea textarea-bordered h-40 mb-4"
        required
        autofocus
    >
{{ attachment.profile.description }}</textarea
    >

It seems like this is the only issue we're running into. I'd expect the formatting to be

        Description of data

    <textarea
        id="description"
        name="description"
        placeholder="Describe the data in the file..."
        class="textarea textarea-bordered h-40 mb-4"
        required
        autofocus
    >
        {{ attachment.profile.description }}
    </textarea>

Here is my config
{
  "plugins": ["prettier-plugin-jinja-template"],
  "htmlWhitespaceSensitivity": "ignore",
  "printWidth": 120,
  "tabWidth": 4,
  "overrides": [
    {
      "files": "*.html.jinja",
      "options": {
        "parser": "jinja-template"
      }
    }
  ]
}

I've messed around with all of these settings and it seems to me that this is an issue with the jinja-template parser.

The output seems to match the default prettier output: https://prettier.io/playground/#N4Igxg9gdgLgprEAuEAeANgQwEZ3QAjCwGdiBeAHRADMIAnAWwFpJY6ICB3J6gV3XRUAfKgAmASwBuhEuSpZcgkCOIAHTFBmZSlEArzCAInGJg641THHR8EavlGYYmVAHo1Gka4mSR8AB7OdHCY+OKiuqImZhZW0FT4UJgMcJHR5pbWUAmqWGBwABYcUXS6xqbmuPgwBXAOTqHimjV11OLocAB03QlE2nIgAUEh1XCBmMGYTNj0JXCi+AVMACwADPgM2CsJwQCOvOLBC5i8MHYQYLzEQgAad-cPj0-PL25DEyFe+uhCFFAgABoQBBMtBiMhQBN2JwAAofKDglCYdCcTAAT3BQOwdEwYAA1nAYABlZJwAAyTTgyGoyOIcCxOPxhKJ6jATQA5sgYHRePSQHSGOIuTy+WNVHBzClYMiACoSpLmEzU2l84gcjoARV4EHgyvQdKBACtiP4ieq4FqdVSkDT9Xz9laYexVIiQNomFA4PN5oDBjj2hyAMIQBgMTDIN0CX1qqDsjoAQRg3PE2FOcBhEopnr1BpABRgDHQAHUCuJ4B58kSEGqrJIy2iI+J2VB6FSgZJeQBJKBRWBEmKWeM9okwNEdHN81TsOlFnGqCNTkwSyRtkBNOl0GBOzDssMToHqOgbiP5wu+qdNGBF8I1ZAARgATKsgXsDsFt7vwzaVUDnNhr6It5IMsQJXHAMo4Iitq5nAmzzFEohkho7K8DucAAGL0GGSYchGJxnCAAC+hFAA
Or am I missing something?

... uh, no. It was I who was missing something :grimacing:


Add note in documentation: V8/Objective-C Threading Limitations

This is not so much a bug but perhaps an inherent limitation to V8/Node/Objective-C FFI bridge.
There are rare scenarios in objective-c that will cause deadlocks.  [NSData dataWithContentsOfURL: ] is one of them.  dataWithContentsOfURL will open a semaphore to block the current main thread while NSURLProtocol will call up the handler for the URL to run.  If (in javascript) you've implemented any types of filters with NSURL or a NSURL Protocol class (for a custom protocol) it will cause a dead lock.
Thread A (NSData) is waiting for a response on the data, it issues a semaphore/mutex to pause the thread; all the while Thread B is NSURLProtocol trying to invoke a javascript function to get the data, however ThreadedCallbackInvokation::WaitForExecution on Thread B sits on a mutex waiting for Thread A to unlock.  Both Thread A and B are waiting for each other.
The symptoms are a spinning beachball of doom when your application hits the call to get any data.
If there's any ideas of how to get around this via node-ffi/etc please chime in. There are usually work arounds in objective-c but I thought i'd post it here incase someone runs into the same problem.

@trevorlinton I assume this is still an issue. Can you post some example code that demonstrates the issue?

@TooTallNate
I can put together a complete one.  An example would actually be a slightly modified version of our unit test here:
https://github.com/TooTallNate/NodObjC/blob/master/test/abstract-class.js
This implements the NSURLProtocol abstract class; although the NSURLProtocol derived class in the unit test returns "NO" for all requests, if it returns "YES" trouble begins to happen.
NSURLProtocol derived classes are called when resources are needed.
If from javascript you issue a blocking (synchronous) request for data, it will completely lock up.  The request is coming from "From JS executes A -> Blocking Apple Obj-C Method -> That Needs To Callback to JS".
The whole thing blows a gasket when the callback discovers the context/isolate is already in use.


The "Front" displays Backend stickers and vice versa

The routes are swapped I guess.
https://strapi-starter-next-ecommerce.vercel.app/categories/front

should show frontend stickers

https://strapi-starter-next-ecommerce.vercel.app/categories/back

should show backend stickers

Thanks, looks this this would be an issue in the strapi-template-ecommerce repo

Thanks, looks this this would be an issue in the strapi-template-ecommerce repo


Feature request: show linter name in location list

I want to see the linter name in the location list in front of every message. It is useful for me so that I can decide whether I like what a given linter is producing, or whether I should disable it. Also, perhaps some warnings I would like to disable, but in order to do that I'd like to know the linter name, so that I can google how to configure it 🙂
I was able to find g:ale_echo_msg_format, but I disable echo for ALE because I use it for other purposes. My primary driver for a linter is the location list.
I'm looking for some kind of g:ale_loclist_msg_format, but I can't find anything like that. Is this possible?

This is related to #852.

This is now supported. See :help g:ale_loclist_msg_format. It was necessary to add support for this option in order to implement the changes in #852, because the codes have to be taken out of text for the items so they can be formatted differently.


Load spawn area chunks on start-up

In order to speed up the initial player join, chunks surrounding the level's spawn coordinates should be loaded in memory at start-up. It would leverage chunkworker and likely be called in the level initialization.

A likely solution is to register an Entity representing the server and then add a chunk hold on spawn chunks for that entity. This will prevent the chunks from being unloaded.
Something like this:
let server_entity = world.create_entity().build();
let mut chunk_holders = world.fetch_mut::<ChunkHolders>();

let view_distance = i32::from(world.fetch::<Arc<Config>>().server.view_distance);
for x in -view_distance..=view_distance {
    for z in -view_distance..=view_distance {
        let chunk = ChunkPosition::new(x, z);
        load_chunk(chunk); // Not sure if this is the actual function, but I think something like this exists.
        chunk_holders.add_for_chunk(chunk, server_entity);
    }
}

In order to speed up the initial player join

Although it's not like joining takes all that long, at least in release mode. The main overhead for join times is chunk sending, not loading, according to my profile tests.


WIP/DNM - Creating a custom interactive examples page for RSP v3 components

✅ Pull Request Checklist:

[ ] Included link to corresponding React Spectrum GitHub Issue.
[ ] Added/updated unit tests and storybook for this change (for new code or code which already has tests).
[ ] Filled out test instructions.
[ ] Updated documentation (if it already exists for this component).
[ ] Looked at the Accessibility Practices for this feature - Aria Practices

📝 Test Instructions:
In RSP check out the interactive examples page.
🧢 Your Project:
RSP

Closing for now.


Bump kernel version to 4.10.30

$Subject

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/12428755324

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/12428755324
Status: success


views_util: Cut import of narrow

(38, 11) → (35, 10). Discussion.

These seem like reasonable new homes, merged, thanks @andersk!


Text/plain to image/vnd.cups-ubrl conversion support?

Hi,
reviewer during Fedora review found out that there is no conversion rule for text/plain document format to image/vnd.cups-ubrl format, but the printing works for the format if pstotiff from hplip is installed.
Should there be a direct conversion support in braille-printer-app as it is for text/plain -> application/vnd.cups-brf, or libcupsfilters+cups-filters will provide necessary conversions to an image document format?

To clarify, printing a text file to brl format works in Fedora 37, before braille-printer-app was made a separate application and uses pstotiff from hplip.  It does not currently work in Fedora rawhide if pstotiff from hplip is installed.

? I don't understand the relation between printing a text file and pstotiff. Does the cups chain happen to convert text to ps then to tiff then to ubrl? That won't be good :)
To make things simpler, we can add a zero-cost brf2ubrl conversion that just translates from brf to unicode braille (basically what iconv -f brf -t utf-8 does)

A low cost conversion would be helpful.   Conversion is text to pdf, ps to tiff then image to ubrl. Sample log below:
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] Applying default options...
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] Adding start banner page "none".
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] Queued on "ubrl" by "fedora".
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] Auto-typing file...
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] Request file type is text/plain.
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] File of type text/plain queued by "fedora".
Feb 19 23:04:26 localhost.localdomain cupsd[869240]: [Job 11] Adding end banner page "none".

Feb 19 23:04:30 localhost.localdomain cupsd[869240]: [Job 11] GPL Ghostscript 9.56.1 (2022-04-04)
Feb 19 23:04:30 localhost.localdomain cupsd[869240]: [Job 11] Copyright (C) 2022 Artifex Software, Inc.  All rights reserved.
Feb 19 23:04:30 localhost.localdomain cupsd[869240]: [Job 11] This software is supplied under the GNU AGPLv3 and comes with NO WARRANTY:
Feb 19 23:04:30 localhost.localdomain cupsd[869240]: [Job 11] see the file COPYING for details.
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] *** WARNING - you have selected SAFER, indicating you want Ghostscript
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] to execute in a safer environment, but at the same time
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] have selected WRITESYSTEMDICT. Unless you use this option with
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] care and specifically, remember to execute code like:
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] \"systemdict readonly pop\"
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] it is possible that malicious code may be able to evade the
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] limited security offered by the SAFER option.
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] Processing pages 1 through 2.
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] Page 1
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] Page 2
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] The following errors were encountered at least once while processing this file:
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] circular reference to indirect object
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] **** This file had errors that were repaired or ignored.
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] **** Please notify the author of the software that produced this
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] **** file that it does not conform to Adobe\'s published PDF
Feb 19 23:04:32 localhost.localdomain cupsd[869240]: [Job 11] **** specification.

Feb 19 23:04:46 localhost.localdomain cupsd[869240]: [Job 11] Ready
Feb 19 23:04:46 localhost.localdomain cupsd[869240]: [Job 11] Set job-printer-state-message to "Ready", current level=INFO
Feb 19 23:04:46 localhost.localdomain cupsd[869240]: [Job 11] PID 869274 (/usr/lib/cups/filter/imagetoubrl) exited with no errors.
Feb 19 23:04:46 localhost.localdomain cupsd[869240]: [Job 11] PID 869275 (/usr/lib/cups/backend/cups-brf) exited with no errors.
Feb 19 23:04:46 localhost.localdomain cupsd[869240]: ubrl benson 11 [19/Feb/2023:23:04:46 +0300] total 0 - localhost test.txt - -
Feb 19 23:04:46 localhost.localdomain cupsd[869240]: [Job 11] time-at-completed=1676837086
Feb 19 23:04:46 localhost.localdomain cupsd[869240]: [Job 11] Job completed.
Feb 19 23:04:47 localhost.localdomain cupsd[869240]: [Job 11] Unloading...

To clarify, printing a text file to brl format works in Fedora 37, before braille-printer-app was made a separate application and uses pstotiff from hplip. It does not currently work in Fedora rawhide if pstotiff from hplip is installed.

But it works when pstotiff from hplip and braille-printer-app is installed... have you restarted cups after hplip installation? cupsd picks up available filters at the start of daemon.

? I don't understand the relation between printing a text file and pstotiff. Does the cups chain happen to convert text to ps then to tiff then to ubrl? That won't be good at all :)

Yes, because there is no text/plain -> image/vnd.cups-ubrl rule in braille.convs :)

To make things simpler, we can add a zero-cost brf2ubrl conversion that just translates from brf to unicode braille (basically what iconv -f brf -t utf-8 does)

That sounds good - IIUC text/plain files will be converted to application/vnd.cups-brf and then to image/vnd.cups-ubrl. It should work, but we have to test it as well.

Make absolutely sure that text gets directly converted do Braille, not rasterized and then the Raster image getting embossed in the low resolution of a Braille embosser, then the text is completely lost. Filter paths doing that need to get avoided under all circumstances.

@tillkamppeter is there a way how to forbid it? I mean if cupsd is able construct a filter chain based on known rules, it will automatically construct the chain, which can convert text into ubrl via tiff. Or is there a way how to disable it without editing rules/removing filter?

The cost of the chains will be enough to avoid the rasterization (which is already given a large cost)


implementation issues

hello, thank you very much to share that peace of art with every body, but still do have issues when i want to implement them on PT because of there dimension, i had to convert them to png and then adjusting the size with paint to fit in PT. but when doing that the image lose its pixels quality, please help me!!
best wishes.

Sorry for being super-slow to respond. If this is still an issue, create a new issue.


Room ids in messages should be clickable

#vector-dev:matrix.org in a message should be seen as hyperlink to the room.

https://github.com/matrix-org/matrix-android-sdk/commit/f3049e4004975258773357e7f9c10429080bef78
https://github.com/vector-im/vector-android/commit/e6c560f1fc562c5c292f5fe552b9d7ea3e2ba2e1


add JPA or spring/micronaut data ORM to supplier service

need an ORM example as it is popular usage. micronaut data also runs on Helidon and supplier service is a good candidate regardless

as discussed we actually want this in the inventory service (for the add/remove/get inventory commands only - not for the inventory check done as part of order message processing) as we will eventually repurpose the supplier service #100

Development completed. Testing is in progress with workshop environment


Add Prefix Caching Benchmark to Serving Benchmark

This PR:

Added a way to benchmark automatic prefix caching using sonnet.txt dataset from llmperf-legacy, with additional arguments to control input, output and prefix lengths.
Fixed TPOT/ITL(time per output token/inter token latency) calculation - now TTFT is excluded from end-to-end latency before latency is averaged over number of generated tokens - 1 (first token).
Misc fixes due to changes in other backends.
Error logging

Not sure if we should separate the prompt generation functions to a different file so it can be reused by benchmark_throughput.py as well. For now I still kept them in the same file.

cc @robertgshaw2-neuralmagic as we discussed offline on benchmarking automatic prefix caching. PTAL and let me know if you have any question!

@tattrongvu - I added some additional functionalities to save error, generated text and actual output len to the result json file so it's easier to debug. I've already tested it myself with num_prompts=200 and request-rate='inf' on a mixtral hosted on 2xA100-80G with vllm 0.3.3, so if you could give this PR a try, that would be great!


Can we add new frontend UI framework in version.2 ?

I've noticed Version.2 for a while and it came out a few days ago. In Version.1 we have codegen, although it lacks documents.
Is it possible to add any other UI Controls to Fabulous instead of using XF?

@edgarfgp has expressed interest in creating a Uno frontend after the MAUI one is finished.
Meanwhile, you can also give https://github.com/fsprojects/Avalonia.FuncUI a try.

@edgarfgp has expressed interest in creating a Uno frontend after the MAUI one is finished.
Meanwhile, you can also give https://github.com/fsprojects/Avalonia.FuncUI a try.

Thanks for your reply. I've tried FuncUI a long time ago.
So it seems that there's no CodeGen in ver.2. And if we want a new UI framework, we should write our own code.

Is it possible to add any other UI Controls to Fabulous instead of using XF?

Yes, Fabulous can support more UI frameworks. Right now, I'm working on adding Fabulous.Maui.
CodeGen v1 was needed since it was the one generating all the core diffing logic for each control, this was an awful lot of code to write by hand otherwise.
In v2, it's not the case anymore. We use a central diffing logic, leaving very little work to do to support new frameworks.
You can see it like that: instead of mapping through JSON, controls are mapped via F# now.
It's not any more difficult, but you can be more fine-grained in how you want your users to use a certain control.
Example of Button:
https://github.com/fsprojects/Fabulous/blob/v2.0/src/Fabulous.XamarinForms/Views/Controls/Button.fs

Is it possible to add any other UI Controls to Fabulous instead of using XF?

Yes, Fabulous can support more UI frameworks. Right now, I'm working on adding Fabulous.Maui.
CodeGen v1 was needed since it was the one generating all the core diffing logic for each control, this was an awful lot of code to write by hand otherwise.
In v2, it's not the case anymore. We use a central diffing logic, leaving very little work to support new frameworks.
You can see it like that: instead of mapping through JSON, controls are mapped via F# now. It's not any more difficult and you can be more fine-grained in how you want your users to use a certain control.
Example of Button: https://github.com/fsprojects/Fabulous/blob/v2.0/src/Fabulous.XamarinForms/Views/Controls/Button.fs

Thanks, ill check the link.


Installing NLog manually (or for combined C++/C# solution)

I have a solution that contains both a C++ and a C# .NET Winforms project.  This seems to cause problems in installing through the NuGet manager (error below).  Is there a way I could install NLog manually?  I've tried referencing NLog.dll, and have added the files NLog.config, NLog.xsd, and although my program compiles and runs, no logfile is created (I'm using log.trace and log.debug)
Type: Question
NLog version: <IP_ADDRESS>
Platform: .Net 4.5
Current NLog config (xml or C#, if relevant)
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <targets>
        <target name="logfile" xsi:type="File" fileName="log.txt" />
    </targets>

    <rules>
        <logger name="*" minlevel="Trace" writeTo="logfile" />
    </rules>
</nlog>

Install-Package NLog -Version 4.3.11
Attempting to gather dependency information for package 'NLog.4.3.11' with respect to project 'PicUSB_Host', targeting 'native,Version=v0.0'
Attempting to resolve dependencies for package 'NLog.4.3.11' with DependencyBehavior 'Lowest'
Resolving actions to install package 'NLog.4.3.11'
Resolved actions to install package 'NLog.4.3.11'
Install failed. Rolling back...
Package 'NLog.4.3.11' does not exist in project '****C++ Project Name ****'
Package 'NLog.4.3.11' does not exist in folder '.....\packages'
Install-Package : Could not install package 'NLog 4.3.11'. You are trying to install this package into a project that targets 'native,Version=v0.0', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.
At line:1 char:1

Install-Package NLog -Version 4.3.11

  + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
  + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand

Well we don't support the Native platform or C++
But if this could be easily solved, let me know.
Is this also fixed with NETSTANDARD in the future?

PS: you could check the internal log if it compiles. See https://github.com/NLog/NLog/wiki/Internal-logging

If you have a solution that contains WinForms-project that uses a C++/Native-project, then you can configure NuGet just to get NLog for the WinForms-project, then just change your C++/Native-project to load and use the NLog.dll manually (without using NuGet).

Cool, which version of NLog.dll it should use?
Current package (NLog 4.3.11)

Using a managed dll from C++ project required CLI/Interop. Unless you experiment with building the managed DLL to have a COM-interface.
You will ofcourse use the NLog.dll that matches the .NET version of the Winforms-project.

Using a managed dll directly from C++ project required CLI/Interop. Unless you experiment with building the managed DLL to have a COM-interface.

NLog 2 was supporting C++ ("NLogC"), but that has been dropped. http://nlog-project.org/2013/12/14/nlog-3-0.html

@zotty is this still an issue?

I assume your question has been answered, if not, let us know!


fix: idle market-service price

Current dependencies on/for this PR:

main

PR #1100   👈

This comment was auto-generated by Graphite.


Validate Formatter Exit Codes

Resolves #216.
Warns on formatter failures, defaulting to <= 0.
Create a utility wrapper helpers.check_exit_code to maintain backwards compatibility.
TODO

logger:warn spawns a blocking notification (i.e. must press enter) and creates a timeout error if not pressed within the default timeout.

Formatter-specific configurations - I suppose I'll need to manually go through and check here?

The format of the log message itself (example below with stylua):

Testing (?)

Type checking - i.e. validating the helper is only ever called with a table or number. However, this helper is meant to be a purely internal function, so I'm not sure if this is necessary.

@mochaaP I can't assign reviewers so please take a look. I know I need to squash commits and a bunch of other stuff but I just wanted to draft the initial API. Let me know everything that can be improved.

@mochaaP I'm going to start going through the formatters one-by-one (pain) to ensure I'm passing the right exit code.

Is there a better way to do this?
I currently make it optional (and default to exit code 0) to indicate formatter success in check_exit_code. Is this fine?

I think (2) is fine.
For (1), we could land the changes first and wait for feedback from ones who actually use them if any problem arises.

Ready for review.

@mochaaP addressing comments above but this is a longer response:
The problem I have with moving the code to the generic factory (and this includes the code duplication comment above) is that I thought this feature only makes sense for formatters.
If we added this to generator factory, wouldn't we also be checking linting exit codes as well?


Documentation (README) give false hope about documentMiddleware.

As explain here : https://github.com/prismagraphql/graphql-yoga/issues/314 documentMiddleware is not yet available. What do you think of removing it from README.md and only document middleware parameter of Graphql-Yoga ?

Yeah, I still wonder why dont we just put the documentMiddleware in this package instead of introducing the yoga or whatever, any reason that documentMiddleware can not be put here?


test-branch/execute-job/wrong-comment-on-pr/e23336d3-ee9b-40ed-a360-661163211e8d

PR for testing 9d0b865b199796d4afebf0311b39bd30833d80c6

comment 1


Revert "lints: remove w_serial_number_low_entropy lint."

There are effectively only three CAs using exactly 64 bit length serial numbers. (Searched with (labels="trusted" and labels="precert" and validation.nss.has_trusted_path=true and not labels="revoked") and parsed.serial_number_hex="????????????????").
Let's considering bringing this length check back, and actually make it trigger a warning on <= 8. There's practically no reason to be riding the "entropy" limit this closely and I think a warning is perfect for that type of lint.

My two cents... any type of "entropy" checks against a single sample will ultimately be error-prone. A high-volume issuer that includes several octets of CSPRNG output beyond the minimum of 8 octets will occasionally generate serial numbers that fall below the minimal bound specified in this lint. Furthermore, any CA that implements controls to manipulate the generated serial number in an attempt to suppress any warnings from the lint will actually have decreased the number of unpredictable bits in the serial number.
Conversely, any CA that includes fixed values in the serial number but otherwise includes exactly 64 bits (or less) of CSPRNG output would not have their certificates flagged by this lint.
The only way to reliably implement this check is to examine a representative sample of a CA's issued corpus of certificates and perform a statistical analysis. As such, I believe this lint would be of negative value as the noise it will generate will be more than any sort of useful insight.

The reason I'm thinking of introducing this lint isn't really to penalize CAs or whatever. It's definitely nothing more than a warning. I'm also planning on making it warn on exactly 64 bit lengths too.
My reasoning is:

This acts as a signal for new CAs to make sure they're not right on the line where it can turn into an incident for them.
There's practically no CA that's using 64 bit or less serials. If they are, they can choose to ignore this lint.

From the three that are, GoDaddy has made a claim in the past that they want to go to 128 bit serials: https://groups.google.com/g/mozilla.dev.security.policy/c/S2KNbJSJ-hs/m/Wnt4QSO9BwAJ
Is there a major reason why we shouldn't have this lint? Maybe we can introduce an "Informational" level lint?

My personal opinion is that the entropy requirements on serial numbers are a defense against bad hash functions, and we've more directly solved that already by removing SHA-1 and MD-5 from the WebPKI ecosystem, and so are not high on my priority list of things to worry about.
There is a strong argument that you can't determine the "entropy" of a serial by examining a single one, for several reasons: Having some number of leading zeros could legitimately happen, so we can't just require it to be greater than 2^64.  Bad RNGs can also generate large-looking serials.  Thus I don't think a lint like this is really going to be effective. Statistical analysis over a large set of serials could be effective, but zlint doesn't seem like the right place for that.

My personal opinion is that the entropy requirements on serial numbers are a defense against bad hash functions, and we've more directly solved that already by removing SHA-1 and MD-5 from the WebPKI ecosystem, and so are not high on my priority list of things to worry about.

Yes, that is why the requirement was introduced. The thing is, that requirement is now in the BRs, and new CAs are much more likely to mess this up when this lint can help prevent a mistake there.

There is a strong argument that you can't determine the "entropy" of a serial by examining a single one, for several reasons:

As mentioned in my comment above, I'm proposing adding this lint back with the understanding that this won't actually measure entropy but act as an "FYI new CA, you're on the line here and there's no reason to be"

Does re-introducing this lint cause any issues?

CONTRIBUTING.md says:

Choosing Result Level. Lints return a single type of status:
...
Warning: Warn can only be used for violations of SHOULD or SHOULD NOT requirements and again should include strong citations. Many certificate authorities block on both Error and Warning lints, and Warning lints should not be used for non-deterministic errors (e.g., calculating whether a serial number has sufficient entropy based on high-order bits.)

Considering that the guidance explicitly calls out what this PR is attempting to do as undesired, I think it shouldn't be added. If folks feels this lint is valuable, it should probably be an INFO- or NOTICE-level lint.

One more thing we see repeatedly in some WebPKI incidents is CAs not understanding the requirements, and assuming the linters will tell them if they're doing something wrong.
Besides the fact that I think this lint is fundamentally not doing what it claims to be doing (measuring entropy), it may give CAs a false sense that they don't need to worry about this requirement.
I would be more supportive of a version of this lint which examines the number of 0 and 1 bits in the serial and fires if there aren't enough of either to be plausibly randomly generated.  That's closer to something that's measuring the number of bits of entropy in a serial, and not just looking if it's "big enough" in a way that's prone to mis-firing.
If there's 64 bits of randomness, I'd expect (off the top of my head) at least 24 1 bits to be set, and a similar (but less, due to the omitted leading 0s) number of 0 bits.  I haven't done the math to figure what the exact numbers are to have a sufficiently low probability of mis-firing on the webpki scale of certificate issuance.

One more thing we see repeatedly in some WebPKI incidents is CAs not understanding the requirements, and assuming the linters will tell them if they're doing something wrong.

This doesn't change with the inclusion, or not, of this lint. We're not close to being able to get to the point where linters can actually enforce every single requirement - however they can act as signals and I still find that valuable for WebPKI. I'm not interested in CAs having incidents to show that they don't understand the requirements. My goal is to make it easier to run a CA with notices of "hey you might be doing something wrong here".
--
Beyond that I do not want to turn this lint into a lint trying to measure entropy. Doing so with a single cert is not useful in my opinion. I'm fine with changing the name to make it even clearer that its not measuring entropy.
@mcpherrinm, if the name of this lint changes - would you be onboard with getting this onto zlint? (Also if you have any name suggestions, I'd appreciate it! Naming things is hard :P)

But we should not be producing warnings for people that meet or exceed the bar.

I've switched this to a notice level lint.
My justification for adding this is as follows:

We've had bugs in the past where EJBCA forced a 64 bit value, by forcing the 64th bit to be constant, effectively reducing this to 63 bits of entropy.
Other CAs may be tempted to do that too, as they might look at the DER output in a serial that is: 0x0012ab... and see that it was encoded with only 7 octets (56 bits)  instead of 8 octects (64 bits) due to how DER encodes integers.
The 3 CAs that are currently in 64 bit mode are, or probably will move away from them anyway. For example, eMudhra has already moved away from 64 bit serials: https://crt.sh/?asn1=13429293975
There's no reason to be generating serials this close to the limit anyway?

Is there a risk you see for adding this lint, as a notice level lint, just to act as a "hey you may be doing something wrong here" notice?

Well, so the problem is that the actual requirement is:
"MUST be a non‐sequential number greater than zero (0) and less than 2¹⁵⁹
containing at least 64 bits of output from a CSPRNG."
Where it was very carefully written that way to avoid any requirements on the encoded value, because a compliant serial number can be extremely small, like for example 42, as long as that happens with probability 2^-64.  That's the danger of trying to technically enforce this particular requirement based on examination of a single certificate.  You can't REALLY say whether a CA complies with this or not without seeing the implementation, though a population of certificates can prove with high probability that the implementation is likely flawed.
I think I can get behind an INFO level lint explaining some of the complexity and danger here, as long as it clearly explains the situation.  I want to meet you in the middle here somewhere.    And I'm sensitive to that because we see a lot of CAs ignoring messages from linters and writing them off as false positives, even though they aren't.  That's why I'm a bit sensitive to linters generating extraneous warnings and information, because it can desensitize users to real issues.

There's no reason to be generating serials this close to the limit anyway

I don't really buy this argument. I think it is valid to use exactly 64-bit random serials, and having a lint that implies otherwise is more likely to lead to workarounds (like the EJBCA 63-bit one) than to fix real problems.
The 63-bit problem is a real one, but it cannot be reliably detected from a single serial, and wouldn't be detected by this lint for the case where the top bit is force-set.
I'm going to open another PR with the idea I'd proposed earlier for counting bits with some analysis showing it has a very low probability of false-positives.

The 3 CAs that are currently in 64 bit mode

I am aware of at least one major CA that generated large serial numbers but contained only 64 bits of CSPRNG output. The non-random octets were seemingly metadata. This was a few years ago and the CA may have since changed its issuance practices, but there are likely more than 3 CAs right at the edge of the 64 bit requirement -- but it wouldn't appear that way by examining just one certificate.

I think I can get behind an INFO level lint explaining some of the complexity and danger here, as long as it clearly explains the situation. I want to meet you in the middle here somewhere.  And I'm sensitive to that because we see a lot of CAs ignoring messages from linters and writing them off as false positives, even though they aren't. That's why I'm a bit sensitive to linters generating extraneous warnings and information, because it can desensitize users to real issues.

@timfromdigicert I agree with what you're saying here. I do not want to cause unnecessary stress & anguish for CAs. I've been on the other side of that coin and I am in complete agreement with you that we don't want this to happen.
To be clear, the vast majority of CAs are using 128 bit sized serial numbers for their certificates. With this change, the chance of this notice happening for a CA that uses 128 bits would mean the first 64 bits of the serial number are 0, and the 65th bit has to be 0 to get a serial number that gets encoded as a 64 bit value. That is a 2^-65 chance of happening. Effectively there is a lower chance of this happening, than getting a collision in a UID space.
I an spend some more time writing godocs inside this linter to ensure that if someone does stumble upon this, it may very well not be an issue.
For CAs that are between 64 and 128 bit serials, the chance of this notice is higher, but with their issuance volume - it is extremely unlikely to hit this condition (For example: the current number of active certificates with serials smaller than 128 bits doesn't add up to half a million)

I am aware of at least one major CA that generated large serial numbers but contained only 64 bits of CSPRNG output.

@CBonnell I'm basing my information off of doing research on censys.io, which may or may not be a complete data set.
I've been using this query (not the best, but works): (labels="trusted" and labels="precert" and validation.nss.has_trusted_path=true and not labels="revoked") and parsed.serial_number_hex="????????????????"

I'm going to open another PR with the idea I'd proposed earlier for counting bits with some analysis showing it has a very low probability of false-positives.

@mcpherrinm This would be a great contribution to this project!

Based on conversation, it doesn't look like there is new support to add this lint compared to past conversations, and I'm going to close.

IMO I think this is still generally valuable, and the response here was mostly stuck around the semantics of "entropy". Per the investigations I had when writing this lint, this lint shouldn't cause errors for existing CAs, and really was designed for new CAs spinning up new infrastructure.


Required oneOf relation

So I'm back to experimenting with the library and trying to implement real-life models with relations.
I see some inconsistency with oneOf relation.
When a model definition has simple fields, the field always contains some data (e.g. empty string for string fields). So the field is kind of required unless explicitly nullable and will not be undefined. Same for manyOf, it just returns an empty array (https://github.com/mswjs/data/pull/201)
But oneOf relation field can have undefined because this field cannot be filled in with some mocked data. As a result, we can have two no data value: null - when the field is explicitly defined as nullable, and undefined when a field is not nullable but a value was not provided when creating an instance.
From my perspective, if a value for oneOf relation is not passed, the creation must fail. Maybe for keeping the existing behavior (can't think about the cases when it is needed, though) it can be done with required attribute for the relation like with unique.
It wouldn't be a big problem as the data is being created by devs in tests, but the resulting TS types for models just don't work with existing types.

Hi, @timkolotov. This is a great proposal. The consistency you describe is the way to go:

oneOf() relationships must have initial value. If not provided, the library should throw. If provided, use the current logic as-is;
If the consumer wishes to have a nullable one-of relationship, describe it explicitly via nullable(oneOf(modelName)).

If marked as nullable and no initial value was provided to the relationship, set its value as null.
If marked as nullable and initial value was provided to the relationship, set its value to the initially provided value.
If marked as nullable, allow updating this relational property to null (i.e. via update etc).

Most of this is already covered by nullable(). The change would be to treat oneOf relationship as non-nullable by default.
Does this sound interesting to you to give it a try? I think the change itself should be rather scoped. I can always help with the code review.

Yep, I'm in. Currently, I just patched the types to shut the TS compiler up, which isn't really safe...

Any update on this issue? Really interested in this as I think I've run into a similar problem. Not sure of the best way around this, with strict mode due to undefined:
// My MSW handler response type
type BasketResponse = {
    basketId: string,
    currency: {
        code: string,
        symbol: string
    }
}

export const db = factory({
    basket: {
        basketId: primaryKey(String),
        currency: oneOf('currency'),
    },
    currency: {
        code: primaryKey(String),
        symbol: String,
    }
})

const currency = db.currency.create()
const basket = db.basket.create({ currency });

type currencyType = typeof currency; // PublicEntity< ... >
type basketCurrencyType = typeof basket.currency; // PublicEntity< ... > |  undefined

const value: BasketResponse = basket // 'Type 'undefined' is not assignable to type '{ code: string; symbol: string; }'.

This makes it a bit awkward to use the basket pulled directly from the db with a MSW handler typed as rest.post<CreateBasketRequest, PathParams, BasketResponse>

@harry-gocity it does look like the same issue indeed. I started working on it but didn't have much time so far. With enough luck I'll have a vacation next week and will be able to dedicate time for this.

@kettanaito finally finished the implementation, please take a look (especially check if the changes in tests make sense to you, some cases I just removed, because they were testing the behavior that doesn't exist anymore).


Added support full named-entity level metrics #458

This PR adds support for strict, exact, partial and entity type matches to NER evaluation.

wanted to make sure that you intended to close this PR. If not I can reopen it

Yes, please reopen it. I have done it by mistake.


Implement the verification of the actual data file

[ ] move the precheck to setup
[ ] Implement the provided template functions
[ ] Implement the verification of the actual data file

Because it's a bit complicated, the implementation of actions (in the third-party dir) will be in the next PR.
You can ignore the files in third-parth dir at this PR.

We need to add source dependency license in the root LICENSE file, refer to https://github.com/apache/skywalking/blob/5445b7d53f1b22b88c5770130f3c97b8855bf17e/LICENSE#L215-L226

We need to add source dependency license in the root LICENSE file, refer to https://github.com/apache/skywalking/blob/5445b7d53f1b22b88c5770130f3c97b8855bf17e/LICENSE#L215-L226

OK. Why the license-eye does not comment on the PR ?

@fgksgf we had a bug in license-eye, https://github.com/apache/skywalking-infra-e2e/pull/8


substrate node template monthly-2021-12 fails to build

The monthly-2021-12 tag of the substrate node template gives this error when attempting to build
error: failed to select a version for `serde_json`.
    ... required by package `sc-cli v0.10.0-dev (https://github.com/paritytech/substrate.git?tag=monthly-2021-12#b6c1c1bc)`
    ... which satisfies git dependency `sc-cli` of package `frame-benchmarking-cli v4.0.0-dev (https://github.com/paritytech/substrate.git?tag=monthly-2021-12#b6c1c1bc)`
    ... which satisfies git dependency `frame-benchmarking-cli` of package `node-template v4.0.0-dev (/Users/justin/dev/new/substrate-node-template-monthly-2021-12/node)`
versions that meet the requirements `^1.0.71` are: 1.0.73, 1.0.72, 1.0.71

all possible versions conflict with previously selected packages.

  previously selected package `serde_json v1.0.68`
    ... which satisfies dependency `serde_json = "^1.0.59"` (locked to 1.0.68) of package `cargo_metadata v0.14.1`
    ... which satisfies dependency `cargo_metadata = "^0.14.1"` of package `substrate-wasm-builder v5.0.0-dev (https://github.com/paritytech/substrate.git?tag=monthly-2021-12#b6c1c1bc)`
    ... which satisfies git dependency `substrate-wasm-builder` of package `node-template-runtime v4.0.0-dev (/Users/justin/dev/new/substrate-node-template-monthly-2021-12/runtime)`
    ... which satisfies path dependency `node-template-runtime` (locked to 4.0.0-dev) of package `node-template v4.0.0-dev (/Users/justin/dev/new/substrate-node-template-monthly-2021-12/node)`

failed to select a version for `serde_json` which could resolve this conflict

Will move this over to the node-template repo, since running the node with this tag is not covered in the existing version of the docs.


Encode any Enums in Routing Header as an Int

See go/clientlibs-routine-headers-numeric-enums for more information.
Enums in every implicit/ explicit routing header should be encoded as an Int.
The Routing Headers seem to be encoded here via a ParamsExtractor.
Example: https://github.com/googleapis/sdk-platform-java/blob/44701e88de88428fb8a94692ffcc85ac815cc203/showcase/gapic-showcase/src/main/java/com/google/showcase/v1beta1/stub/HttpJsonComplianceStub.java#L850-L855
This should be modified to get the Int value instead.
Generator changes:
Affect the AbstractTransportServiceStubClassComposer. Specifically around: https://github.com/googleapis/sdk-platform-java/blob/44701e88de88428fb8a94692ffcc85ac815cc203/gapic-generator-java/src/main/java/com/google/api/generator/gapic/composer/common/AbstractTransportServiceStubClassComposer.java#L1252-L1258
Both methods should be modified so that the changes are applied for both implicit and explicit routing headers.

Created a new issue for implicit routing headers case -  https://github.com/googleapis/sdk-platform-java/issues/2368 .

FYI, only implicit routing headers can be fixed currently because explicit routing headers are implemented as strings by design. See this doc for more details.

Implicit Routing Headers were implemented. Discussion is pending explicit routing headers and how to properly implement it.


QP problem - different solution with CLARABEL and OSQP

Hi,
when trying to solve this QP problem
import numpy as np

m = [10_000, 10_000, 10_000, 10_000]
s = [0.4, 0.6, 0.9, 0.1]
weights = [20_000, 20_000, 20_000, 20_000]
dt = 15.0/60
n = len(m)

x = cp.Variable(n)
total_m = 30_000
constraints = [
    0 <= x,
    x <= m,
    cp.sum(x) == total_m,
]

objective = cp.sum_squares(s + x * dt / weights - 0.5)

problem = cp.Problem(cp.Minimize(objective), constraints)

for solver in [cp.CLARABEL, cp.OSQP]:
    print(f"SOLVER: {solver}")
    problem.solve(solver=solver)
    print(f"{problem.value = }")
    print(f"{problem.status = }")
    # print(f"{problem.solver_stats = }")
    print(f"{np.round(x.value, 1) = }")

I get these solutions
SOLVER: CLARABEL
problem.value = 0.28687500022009166
problem.status = 'optimal'
np.round(x.value, 1) = array([10000., 10000.,     0., 10000.])
SOLVER: OSQP
problem.value = 0.3743074440687389
problem.status = 'optimal'
np.round(x.value, 1) = array([7510., 7490., 7460., 7540.])

It seems that CLARABEL was able to find a better solution on the boundary. Is this an expected behaviour?

The square (dt/weights)**2 is of order 1e-10 so the quadratic coefficients in the QP representation of the problem basically disappear. You can see that if you solve for the objective  cp.sum_squares(np.array(s) - 0.5) you will get almost the same answer.
Conic solvers do not show this behavior because for them in conic form the expression inside sum_squares is passed directly in its affine form, so the smallest coefficients are only of order dt/weights, that is 1e-5.
So I would argue that your problem is just too badly scaled when cast as a QP: the quadratic coefficients are vanishingly small. The solution would be to rescale your m,total_m,weights,x down by 1e+3 or so (that would be a beneficial and reasonable modeling strategy regardless of solver).

Oh, indeed. Thank you for quick response.


DCI Working Group: Inclusive Naming Proposal

Detailed proposal can be found here https://wiki.hyperledger.org/display/TSC/DCI+Working+Group%3A+Inclusive+Naming+Proposal

Update: I've added the fresh links to the proposal for DCI Lint which now show the screencast tutorial on how to introduce automatic pull request DCI linting for any HL or HL Labs project.

Approved by voice vote on 17 FEB 2022


can i server my model with triton inference server

❓ Question

What you have already tried

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

PyTorch Version (e.g., 1.0):
CPU Architecture:
OS (e.g., Linux):
How you installed PyTorch (conda, pip, libtorch, source):
Build command you used (if compiling from source):
Are you using local sources or building from archives:
Python version:
CUDA version:
GPU models and configuration:
Any other relevant information:

Additional context

after torch-tensorrt complie

We are currently working on integration with Triton's LibTorch backend so this workflow is supported out of the box.
cc: @borisfom

Can we serve it with tensorrt backend in Triton ? I tried converting to trt engine using the following code.
trt_ts_engine = torch_tensorrt.ts.convert_method_to_trt_engine(traced_script_module,
                                                               method_name='forward',
                                                               inputs=inputs,
                                                               truncate_long_and_double=True)
with open(f"{OUT_PATH}/model.plan", 'wb') as f:
    f.write(trt_ts_engine)

But I get this "Version tag does not match" error in Triton.
E1213 11:30:16.510379 1 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 0)
E1213 11:30:16.510406 1 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.)
E1213 11:30:16.646657 1 model_repository_manager.cc:1186] failed to load 'rbg_720_outer' version 1: Internal: unable to create TensorRT engine

Using: TensorRT version: <IP_ADDRESS> Triton Inference Server container image, release 21.11

Its working for me with fp32/fp16 so far (nvcr.io/nvidia/pytorch:21.11-py3 and nvcr.io/nvidia/tritonserver:21.11-py3 images)

Any updates? Is this working yet on triton server?


Download image using contentURL in SDK Python

@joshamil ,
MS teams framework treat attachment object and clipboard image differently.

When user paste an image in MS teams chat, it does not treat it as an usual attachment. Rather it receives as an image type and stores it in MS Teams internal storage and sends the Image URL in Activity.Attachment[0].
you need to download the image content from that URL and upload to your blob storage.
MS Teams attachment URLs are secured by a JWT Token, so we need to pass the token from our bot to access it.

Here is a sample c# code to get the image form MS Teams URL and save it to solution file. You can add this logic in ActivityHandler method.
if (turnContext.Activity.Attachments != null && turnContext.Activity.Attachments.Count > 1
            && turnContext.Activity.Attachments[0].ContentType=="image/*")
        {
            var attachment = turnContext.Activity.Attachments[0];
            using (HttpClient httpClient = new HttpClient())
            {
                // MS Teams attachment URLs are secured by a JwtToken, so we need to pass the token from our bot.
                var token = await new MicrosoftAppCredentials("appId", "password").GetTokenAsync();
                httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
                var responseMessage = await httpClient.GetAsync(attachment.ContentUrl);
                var contentLenghtBytes = responseMessage.Content.Headers.ContentLength;
                // You could not use this response message to fetch the image for further processing.
                if (responseMessage.StatusCode == System.Net.HttpStatusCode.OK)
                {
                    Stream attachmentStream = await responseMessage.Content.ReadAsStreamAsync();
                    attachmentStream.Position = 0;
                    var image = System.Drawing.Image.FromStream(attachmentStream);
                    image.Save(@"ImageFromUser.png");
                    await turnContext.SendActivityAsync($"Attachment of {attachment.ContentType} type and size of {contentLenghtBytes} bytes received.");
                }
            }
        }

Originally posted by @Subhasish-msft in https://github.com/MicrosoftDocs/msteams-docs/issues/1561#issuecomment-615186022

Need above solution in Python. Facing similar issue

Hello @yashna-shravani - Thanks for reporting your issue.
We will look into it and get back to you.

Any update here? How can I authenticate using appID and password to download content from contentUrl using Python? @ChetanSharma-msft

If you are implementing your own authentication, then please follow this document - https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication?view=azure-bot-service-4.0
If you are using Bot builder Python SDK, you need to use MicrosoftAppCredentials Class and call the get_access_token() method to fetch the token. Please refer - https://github.com/microsoft/botbuilder-python/blob/d02f0c40ccd505927c56ad6a434ccab2e6a13a0c/libraries/botframework-connector/botframework/connector/auth/microsoft_app_credentials.py#L12
Then you can pass the above token to fetch the contentUrl.

Thanks!

we are closing this issue, please feel free to reach out to us if you have any questions.


Input Definition: input data endpoint

Create an endpoint to accept an Input data in JSON or CSV format. This is part of the proposal described in #538. For phase one, we will be very strict on data format; if the data does not match the input definition exactly then we raise an error.
/index/idx/input/input-definition
idx is the name of the index that the Input Definition is applied to
input-definition should match an Input Definition already registered with the cluster. If the Input Definition specified does not exist we should return an error.
Send data as a json object:
POST /index/idx/input/taxi-ride -d "<json>"
Content-Type: application/json

Send data as a csv record:
POST /index/idx/input/taxi-ride -d "<csv>"
Content-Type: text/csv

Only handle JSON data in the ticket.


Python: support Django 2.x and 3.x

I think it's done, but submitting as draft so I can look it over once more tomorrow (and fix any tests that could have failed)

Oh wait, I actually wanted to keep this as a draft for a bit more, since I wanted to run it against real projects to see if it works :wink:

So I managed to look at a real project, and once django is available in the analysis, the new analysis does work on https://github.com/jumpserver/jumpserver, that uses django 2.1.11.
We find 376 django routes, but only for 46 of them we successfully identify at least one hander function (class-based views can have more than one handler function for a single route). So I won't say that everything is perfect, but it's a start :muscle:


Do the examples really work?

Hi,
I'm trying to figure out how Wuff works and how I can use it for my company build system. I've a very small knowledge of Wuff. So, I think building the examples could be a good way to start learning Wuff.
If I run the following command from the wuff folder:
./gradlew -b examples/PluginXml-1/MyEclipsePlugin/build.gradle

I get the BUILD SUCCESSFULL message, great! But what's not great is nothing has been really built. In the build folder I only get a maven-ant-tasks-2.1.4-SNAPSHOT.jar which is obviously not the plugin I'm trying to build.
I've tried with other examples from the ones provided, and that's the same. Any idea?

I think you've missed the task name (build) in the gradle call. Try:
./gradlew -b examples/PluginXml-1/MyEclipsePlugin/build.gradle build

You should see the output in wuff-0.0.18\examples\PluginXml-1\MyEclipsePlugin\build

Obviously!  Thankd for the help.


Add typings for typescript

Hello there!
I like the library and would love to contribute by adding typescript support.
Added index.d.ts with typings for this library.

@Naturalclar Thank you!
looks good, I'm approving the PR.


Creating product planning draft

Refer the the guidelines:
https://docs.google.com/document/d/19J8In7yXjVMQPCNsysOrMUZcK1h6n60V37eOJP-xMWE/pub

@YDutchy is editor / has the final responsibility
The way I view this:
This week

@mvanmeerten and @YDutchy create a draft
everyone checks to see whether they agree with that description of our planning
@mvanmeerten and @YDutchy finish the document
we hand in the draft

next week

we get feedback
someone (to be decided for next sprint) finishes the document


Update CI to use packages

Update CI to use packages

https://github.com/Mergifyio backport humble iron


DOC: improve stats tutorial documentation

What does this implement/fix?
Just a quick fix in the stat tutorial as a colleague had trouble with the stat_fun throwing a cryptic error if one used A*B in the ANOVA function.
Additional information
Any additional information you think is important.

Thanks @jasmainak


perf(core): improve binary size with api enum serde refactor

What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Docs
[ ] New Binding issue #___
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[x] Other, please describe: performance improvement

Does this PR introduce a breaking change?

[ ] Yes, and the changes were approved in issue #___
[ ] No

Checklist

[ ] When resolving issues, they are referenced in the PR's title (e.g fix: remove a typo, closes #___, #___)
[x] A change file is added if any packages will require a version bump due to this PR per the instructions in the readme.
[ ] I have added a convincing reason for adding this feature, if necessary

Other information
This was discussed with @chippers and @JonasKruckenberg

Basically this changes the allowlist error message from the runtime to the deserialization step. The binary is 200KB smaller now (release build, stripped).

I'll merge this to a feature branch because there's more improvements to push.


Surface Pro 3 driver does not download

Info: Selecting known model Surface Pro 3		01-01-1601 00:00:00	0 (0x0000)
Info: Selecting known model Surface Pro 4		01-01-1601 00:00:00	0 (0x0000)
Info: Selecting known model Surface Pro 5		01-01-1601 00:00:00	0 (0x0000)
Info: Validating all required selections have been made		01-01-1601 00:00:00	0 (0x0000)
Validation state is True		01-01-1601 00:00:00	0 (0x0000)
Info: Importing Into Products: ConfigMgr - Standard Pkg		01-01-1601 00:00:00	0 (0x0000)
Info: Download Type: Drivers		01-01-1601 00:00:00	0 (0x0000)
Pre-Check: Download path set To \server\source$\OSD\Drivere 2\Temp		01-01-1601 00:00:00	0 (0x0000)
======== Starting Download Processes ========		01-01-1601 00:00:00	0 (0x0000)
Info: Models selected: Microsoft Surface Pro 3		01-01-1601 00:00:00	0 (0x0000)
Info: Operating System specified: Windows 10		01-01-1601 00:00:00	0 (0x0000)
Info: Operating System architecture specified: x64		01-01-1601 00:00:00	0 (0x0000)
Info: Site Code specified: P03		01-01-1601 00:00:00	0 (0x0000)
Info: Download Path specified: \server\source$\OSD\Drivere 2\Temp		01-01-1601 00:00:00	0 (0x0000)
Info: Package Path specified: \server\source$\OSD\Drivere 2		01-01-1601 00:00:00	0 (0x0000)
Info: Windows 10 build lookup required		01-01-1601 00:00:00	0 (0x0000)
Info: Windows 10 build 10.0.19041.1 and version 2004 identified for driver match		01-01-1601 00:00:00	0 (0x0000)
======== Processing Microsoft Surface Pro 3 Downloads ========		01-01-1601 00:00:00	0 (0x0000)
Info: Starting Download, extract and import processes for Microsoft model: Surface Pro 3		01-01-1601 00:00:00	0 (0x0000)
Info: Setting Microsoft variables		01-01-1601 00:00:00	0 (0x0000)
Info: Atteming match for SurfacePro3		01-01-1601 00:00:00	0 (0x0000)
Info: Microsoft AKA shortlink URL is https://aka.ms/SurfacePro3/19041		01-01-1601 00:00:00	0 (0x0000)
Info: Microsoft redirected URL discovered is https://www.microsoft.com?ref=aka		01-01-1601 00:00:00	0 (0x0000)
Info: Creating custom package root folder - Drivers\ 		01-01-1601 00:00:00	0 (0x0000)
Info: Creating custom package subfolder - Drivers\2004\ 		01-01-1601 00:00:00	0 (0x0000)
Info: Remaining models to process: 0		01-01-1601 00:00:00	0 (0x0000)
======== FINISHED PROCESSING ========		01-01-1601 00:00:00	0 (0x0000)

Windows 10 2004 drivers for the Surface Pro 3 for example is not currently available (https://www.microsoft.com/en-us/download/details.aspx?id=38826). The tool will attempt to discover the AKA redirection URL, but this is depending on availability from MS


Controllers without PathAttribute throw an exception

There is an error in the header. Doesn't really work if there are no attributes at all


Get KS version used for sorted data asset

As there will soon be a mix of assets run with different versions of kilosort,
we'll need to be able to identify them easily.
The version of kilosort used was previously encoded in the asset name, but that no
longer seems to be the case.

[x] Add a staticmethod to the ecephys extension which returns kilosort version and
works for any sorted data asset ever made.

@alejoe91: I'm hoping you can guide on all the ways of doing this for the different versions of the pipeline

This should probably return sorter_name instead..
I think I've found all the places it could live and tested on all assets in docdb:
https://github.com/AllenNeuralDynamics/aind-session/commit/75d8bb52fd0c29b29aa72ace6817c865d02790cd


Management tooling - Job listing

For async jobs like generating reports, that take some time to create by evitaDB, we will create separate evitaLab-wide job listing viewer, where all these "jobs" will be listed. The listing will be unified for different types of jobs. The evitaLab will provide API for registering such jobs with monitoring etc. Jobs will be usually some reports to generate or backup/restore catalog.

[ ] internal model for registering jobs
[ ] internal API for registering jobs
[ ] UI for registered jobs

Some tasks provide progress information in %. Some tasks will provide a file as a result.

@novoj this will need some backing in evitaDB itself, idk if it's good idea to implement this solely on FE due to the possible page reloads (it can be probably solved by browser storage, but we need to discuss it).

What do you have on mind exactly?

[ ] implement job type into job listing

Implemented fully in dev.