Exception in thread "Thread-1798" redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: recv failed

Iam checking Redis Read and Write with Concurrent Users at A time(Multi-threading).it is working fine upto 380 Users and then it throws above Exception.
Any help greatly appreciated.
Thanks.

@Pratap6
Hello.
Could you share your snippet which part is using Jedis?
I've not seen that message and haven't reported.
If you use JedisPool (and you should in a multithreaded env.) could you configure GenericObjectPoolConfig to test before borrow?

@Pratap6 Please describe Jedis version, Redis version, single/multi threaded also.

@HeartSaVioR My versions are jedis 2.6.2 and redis-2.8.7 and commons-pool2-2.3.jar,
I'm attaching the class files for Reading, Writing and Jedis Connection SingleTon Classes.
Any help to resolve this issue.
Thanks in advance.

Reading From Redis:
public class MyAppRedisReadClient {
public static void main(String[] args) {
String t1 ="7:spuser6:6:e5cc606d7f00000113159de7b49f89e9:6:Demand by Geo";
String t2 =":CAM~~TW";
for (int i = 0; i < 400; i++) {
List keysList = new ArrayList();
String key =null;
Runnable thread=null;
Executor e=null;
for (int j = 0,k=1; j<2500; j++) {
key = t1+j+t2;
keysList.add(key);
if(keysList.size()==500){
thread=new MyAppRedisReader("Thread-"+i+"-"+k,keysList);
new Thread(thread).start();
keysList=new ArrayList<>();
}	}	} 	}  }
public class MyAppRedisReader implements Runnable {
private List keysToRead;
public static JedisPool jedisPool;
public MyAppRedisReader(String name, List keysToRead) {
this.name = name;
this.keysToRead = keysToRead;
}
@Override
public void run() {
List list = new ArrayList<>();
try (Jedis jedis = JedisConnectionManager.getInstance().getJedis()) {
long pollStartTime = System.currentTimeMillis();
if (keysToRead != null) {
for (Iterator iterator = keysToRead.iterator(); iterator.hasNext();) {
Set set = new HashSet<>();
String key = iterator.next();
set = jedis.zrangeByScore(key, "-inf", "+inf");
list.add(set.size());
}
} }	catch (JedisConnectionException e) {
System.out.println(e + "....... : fail to get connection...");
}finally {
// jedisPool.returnResource(jedis);
}
}
}

Jedis Connection SingleTon Class
Jedis Connection SingleTon Class:
public class JedisConnectionManager {
private static  JedisConnectionManager instance =null;
private static JedisPool pool;
private JedisConnectionManager() {
connect();
}
public final static JedisConnectionManager getInstance() {
if (instance == null) {
synchronized (JedisConnectionManager.class) {
if (instance == null) {
instance = new JedisConnectionManager();
}
}
}
return instance;
}
private void connect() {
        // Create and set a JedisPoolConfig
        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxWaitMillis(8000);//10000
        poolConfig.setTestOnBorrow(true);
        poolConfig.setTestOnReturn(true);
        poolConfig.setTestWhileIdle(true);
        poolConfig.setNumTestsPerEvictionRun(50);
        poolConfig.setMinEvictableIdleTimeMillis(60000);
        poolConfig.setTimeBetweenEvictionRunsMillis(50000);
        poolConfig.setMaxTotal(2000);
        poolConfig.setMaxIdle(150);
        poolConfig.setMinIdle(100);
        pool = new JedisPool(poolConfig, "host", 6379, 0);
        }
    public void release() {
        pool.destroy();
    }
    public Jedis getJedis() {
    	StackTraceElement[] stacktrace = Thread.currentThread().getStackTrace();
    	StackTraceElement se = stacktrace[2];//maybe this number needs to be corrected
		Jedis jedis = null;
		int cnt = 1;
		do {
			try {
				cnt++;
				if(cnt>=1000) {
					System.out.println("Not getting Jedis even after "+cnt+" Attempts.");
					return null;
				}
				jedis = pool.getResource();
			} catch (Exception e) {
				System.err.println("("+cnt+") Got exception during getting jedis connection. "+e.getMessage()+" Called from "+se.getClassName()+"."+se.getMethodName());
				if(cnt>3) {
					e.printStackTrace();
				}
			}
		} while(jedis==null);
		if(cnt>2) {
			System.out.println("Got Jedis after "+cnt+" Attempts.");
		}
		return jedis;
    }

    public void returnJedis(Jedis jedis) {
        pool.returnResource(jedis);
    }

}

@Pratap6 Could you please attach your source code into Gist if you don't mind?

@HeartSaVioR, As per your request, I have shared my source code into Gist, could you please verify and help me to resolve this issue.
Thanks in advance.

@Pratap6 Please leave your gist link.

@HeartSaVioR Please find my source code link below:
https://gist.github.com/Pratap6/a7c2903eeb3f8757ec51

@Pratap6 I've just run your Writer and Reader code in my computer and I didn't get any errors.
I've tested it with latest Jedis code (master branch) and with latest Redis (unstable branch) code.
Can you give it a try with that setup so we can start narrowing down the problem?

@Pratap6
I skimmed your gist codes but cannot find any issue.
One different environment would be commons-pool2-2.3.jar. Could you change your project's commons pool with 2.0 and try again?

@marcosnils Thank you for quick response, If you increase concurrent users in Writer class (300+)users and Reader Class(400+) users,then at that time I'm getting those 2 errors.
Writer Class:
public class MyAppRedisWriteCilent {
public static void main(String[] args) {
//Increase ConCurrent Users here --- 300+ users above
for (int i = 0; i <300; i++) {
Runnable thread = null;
for (int j = 0; j < 2500; j = j + 500) {
thread = new MyAppRedisWriter("Thread-" + i, j, (j + 500));
new Thread(thread).start();
}
}
}
}
Reader Class:
public class MyAppRedisReadClient {
public static void main(String[] args) {
String t1 ="7:spuser6:6:e5cc606d7f00000113159de7b49f89e9:6:Demand by Geo";
String t2 =":CAM~~SW";
//Increase ConCurrent Users here --- 400+ users above
for (int i = 0; i < 380; i++) {
List keysList = new ArrayList();
String key =null;
Runnable thread=null;
Executor e=null;
for (int j = 0,k=1; j<2500; j++) {
key = t1+j+t2;
keysList.add(key);
if(keysList.size()==500){
//System.out.println("Size:"+keysList.size());
thread=new MyAppRedisReader("Thread-"+i+"-"+k,keysList);
new Thread(thread).start();
keysList=new ArrayList<>();
}
}
}
}
}
Thanks in advance.

@marcosnils As per your suggestion I've tested with new setup having latest Redis-2.8.19 stable version and Jedis-2.6.2. I got those 2 problem's again.
At Writer Class, I have tested with 400+users I got the below error:
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: recv failed : fail to get connection
At Reader Class, I have tested with 420+ users I got the below error:
Got exception during getting jedis connection. Could not get a resource from the pool Called from info.redis.test.MyAppRedisReader.run
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:50)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:88)
at info.redis.test.JedisConnectionManager.getJedis(JedisConnectionManager.java:59)
at info.redis.test.MyAppRedisReader.run(MyAppRedisReader.java:27)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at redis.clients.util.Pool.getResource(Pool.java:48)

@HeartSaVioR
As per your suggestion, I have changed that pool into commons-pool2-2.0.jar and tested again. I got the same error.

@Pratap6 how do you exactly run your test?, do you run both Writer and Reader at the same time or one after the other?. I've just increased the values to 400 for the Writer and 420 for the Reader and I didn't get any errors at all.
The way I'm testing is running first the WriterClient and then the Reader. I've also used latest redis stable version and Jedis master branch.

@Pratap6 can you try increasing the pool MaxTotal to 5000 in JedisConnectionManager, try again and share your results please?
Also which Java version and OS are you performing your tests on?

@marcosnils
Thanks for your quick reply,
I'm testing One after the other. As per your suggestion I have increased pool MaxTotal to 5000 and tested again, Same errors are coming.
Java Version: 1.7.0_65
OS:  Ubuntu 14.04 LTS

@Pratap6 do you have some custom configuration set in your sysctl.conf file?. Are you using the default redis.conf in your server?
"recv failed" usually happens when someone (either the OS or Redis) closes the connection to the Java client.

@marcosnils
Yes I'm using default redis.conf in my server, I haven't made any custom configuration in sysctl.conf file. Could you please help me what are the required changes needed in sysctl.conf file.
And how to resolve this issue -- recv failed error
Please suggest me on this.
Thanks.

@Pratap6 I'll give it a look later today and let u know if I can find something.  @HeartSaVioR do you have any ideas what may be happening?, have you tried running the code in your computer?

@marcosnils No, I'll give it a try.

@marcosnils I ran Writer and Reader at same time, and receive some error messages.
(Also modified code to always print stack trace)
(2) Got exception during getting jedis connection. Could not get a resource from the pool Called from test.MyAppRedisReader.run
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
	at redis.clients.util.Pool.getResource(Pool.java:50)
	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:88)
	at test.JedisConnectionManager.getJedis(JedisConnectionManager.java:58)
	at test.MyAppRedisReader.run(MyAppRedisReader.java:23)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException: Unable to validate object
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:497)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
	at redis.clients.util.Pool.getResource(Pool.java:48)
	... 4 more

JedisFactory returns false without any messages if it cannot validate object so I can't follow up.
I'll try to modify JedisFactory to print stack trace if it throws and try again.
@Pratap6
Could you please check Redis log and paste it if Redis wrote error log?
If you see errors from Redis log like below, please try to increase max open files.
883:M 28 Jan 23:42:53.299 # Accepting client connection: accept: Too many open files in system

@marcosnils
I found that there're some 'connection reset' in validating object.
But I have no idea about causing socket to broken.
@Pratap6
Btw, your ReadClient requests a lot of objects from pool, worst scenario is 380 * 2500 = 950000.
Jedis should create sockets too fast, and Redis should handle your 950000 requests and relevant 950000 * N PINGs (testOnBorrow and testOnReturn, and testWhileIdle) in a short time.

@HeartSaVioR
I haven't find any errors in my Redis log file. please find my redis.log file
.

@Pratap6 Could you please disable THP and try again?

@HeartSaVioR
Hi,I tried with below command to disable THP in my Environment.
$ echo never > /sys/kernel/mm/transparent_hugepage/enabled
After that I have tested Redis Write and Read one after another, again same errors are coming.
Thanks,
same errors are

@HeartSaVioR If you run Write first and then Read, do you get the same errors @Pratap6 is getting?
Best!.

@marcosnils
I didn't get 'Software caused connection abort: recv failed'. I got another errors, which can be seen easily.
(broken pipe, connection reset, etc.)

@Pratap6
Please refer http://unix.stackexchange.com/a/99172 for disabling THP completely.

@HeartSaVioR
Thanks for the quick reply, As per above link I have disabled THP completely in my Environment.
And then Rebooted my machine and started my Redis server.
After that I tested Redis Write with 400 users, I got same error after disabled THP.
(2) Got exception during getting jedis connection. Could not get a resource from the pool Called from info.redis.test.MyAppRedisWriter.run
Got Jedis after 3 Attempts.
Got Jedis after 3 Attempts.
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: recv failed : fail to get connection...
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: recv failed : fail to get connection...

@Pratap6 can you change the redis loglevel to "debug" and check if something pops out?

@Pratap6
Could you increase /proc/sys/net/core/somaxconn to higher value, 1024 or 32768?
Somaxconn is described by https://computing.llnl.gov/linux/slurm/high_throughput.html
and you can change it, refer http://serverfault.com/a/271386 for help.

@HeartSaVioR
As you suggested, I made changes in /proc/sys/net/core/somaxconn file ,but Redis Write is working upto 400 users only(when I check with 500 users, It is not working. I got same errors as mentioned above)
And Redis Read(I run with 420+ users) takes lots of time after 120 users(but still program is running).
Is there any Ubuntu environment changes are needed,please suggest me.
Thanks in advance.

@marcosnils
As you suggested, I have Run the Redis in Debug mode, I haven't find any issue in Redis Logger file.
I found this one in log file:
[2571] 30 Jan 12:19:13.103 - Reading from client: Connection reset by peer
[2571] 30 Jan 12:19:13.103 - Reading from client: Connection reset by peer
Is this Issue?
Thanks in advance.

@Pratap6

Did you reboot to take effect?
Connection reset by peer : I don't know about Network deeply, but this SO answer can help you.
http://stackoverflow.com/a/1434506
Try poolConfig.setMaxIdle() to 2000, and see it helps. (Invalidated socket will be removed by "test on borrow" and "test while idle")

Btw, I'm wondering what scenario needs initiating a hundred thousand threads (400 * 2500 = 100,000) in a row. Actually I'm really wondering why it starts 2500 threads when user comes in.
Initializing thread, starting thread, stopping thread, context switching between threads, initializing socket, closing socket, all things has overhead.
I'll be honest with this issue, issue may not resolved unless you take another approach.

@HeartSaVioR
It seems something related to Heap. I have installed Redis in 32 GB RAM machine and now that issue is not occurring. Please suggest ideal H/W requirement for Redis and also help us to get location of configuration files where VM and Heap memory for Redis can be updated.

@Pratap6 are you refering to the JVM Heap?. I'm totally confused when you're asking for Redis hardware specs when Jedis is coded in Java. Can you please give some more arguments why you believe it's the Heap and what exactly do you mean by Heap?

@HeartSaVioR
@marcosnils , Sorry for the confusion. It's regarding to JVM Heap only. I exported my project as a executable jar file and tested above scenario. here I was succeeded upto 700 concurrent users there is no pool exception. After that when I increase the users I'm getting Pool Exception...
My current scenario is, If A user send a request to Web service, then we get response as list of keys(Example: List contains 2500 keys), then the response list is splitting into 5 parallel processing (ex: 5 parallel process --> each list having 500 keys), then I'm doing Read & Write operations on Redis with concurrent users (Creating huge threads).NOW I have succeeded upto 700 users with your support and suggestions.
Please suggest me any another approach to resolve my problem.
Best!.

@Pratap6 and how did you come to the conclusion that the problem was the JVM heap?. Increasing the RAM of the underlying system doesn't modify the JVM heap automatically. Did you manually tune up the Xmx and Xms parameters into your program to increase the amount of users?
Regarding to optimizations for your current problem (if you only have one redis server) I believe that by using sharding and pipelining you should get the response times and throughput expected. How many keys are you supposed to handle in a worst case scenario?.
Redis is extremely fast, I've seen it handle up to 30k queries per second through the network for a single thread application.

hi @Pratap6 any updates about this?

hi @marcosnils ,sorry for the late response.
I have tested upto 800 users with a single redis instance with my jedis client code.
Thanks for your support @HeartSaVioR and @marcosnils .
Very good support...


The software is not working on Big Sur 11.5.1 with Monitor ASUS VA325

Checklist

[ x ] I have searched for existing issues
[ x ] I have looked through the wiki
[ x ] I have updated MonitorControl to the latest version

Describe the bug
Screen turn black and lose both sound and image
To Reproduce
Install with the
brew install --cask monitorcontrol
Downloading https://github.com/MonitorControl/MonitorControl/releases/download/v2.2.0/MonitorControl.2.2.0.dmg
And configure the permisson allow app change system
Run Find and type in monitorcontrol to run
Expected behavior
It could control the volume and lightness of monitor
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Environment Information (please complete the following information):

macOS version: Big Sur 11.5.1
Mac model: MacBook Pro (13-inch, 2019, Four Thunderbolt 3 ports) 2.4 GHz Quad-Core Intel Core i5
MonitorControl version:
Monitor(s): ASUS VA325 with HDMI
Apple Silicon/M1 (yes or no): no

Hi @chenzhendong - the v2.2.0 was pulled due to various issues. Do you have this with 2.1.0 or 3.0.0 as well? Thanks!

I'll close this issue for now but please feel free to comment if there is anything we can do. If the issue persists, we can reopen this issue! Thank you!

I just had this issue after downloading the app. It recognized both of my HP E243 displays and then i adjusted one of the displays to 0. Now it won't come back on!! It won't recognize the signal even after pulling out the cord, as if it's bricked!

after unplugging my monitor and unplugging my computer, my screen is responding to input signal again.


Python 3.12 compatibility

First of all, thank you for the awesome package and all your contributions so far! I'd like to request that support for Python 3.12.x version be added to the scrapergraph-ai package.

hi, we have some dependencies that do not work with python 3.12, for the moment we cannot do that.
Have a nice day


workspaceFolder stripped of separators. Add escaped variant of workspaceFolder?

Does this issue occur when all extensions are disabled?: N/A

VS Code Version: 1.70.2
OS Version: Windows 10

I have an example here where I need to pass a path to an elf file in gdb: https://github.com/ChrisIdema/nrf_blinky_stlink/ . This is needed in both the launch.json and task.json.
My launch.json file contains a configuration with the argument setupCommands. This argument needs absolute paths. I cannot use ${workspaceFolder} since that one isn't escaped in Windows. It works on Linux.
Here are paths that work:
{ "text": "-file-exec-and-symbols C:\\\\git\\\\nrf_blinky_stlink\\\\blinky\\\\build\\\\zephyr\\\\zephyr.elf", "description": "load file", "ignoreFailures": false},
{ "text": "-file-exec-and-symbols C:\\\\\\\\git\\\\nrf_blinky_stlink\\\\blinky\\\\build\\\\zephyr\\\\zephyr.elf", "description": "load file", "ignoreFailures": false},
{ "text": "-file-exec-and-symbols C:/git/nrf_blinky_stlink/blinky/build/zephyr/zephyr.elf", "description": "load file", "ignoreFailures": false},
{ "text": "-file-exec-and-symbols C:/git${pathSeparator}${pathSeparator}nrf_blinky_stlink/blinky/build/zephyr/zephyr.elf", "description": "load file", "ignoreFailures": false},

Here are paths that don't work:
{ "text": "-file-exec-and-symbols ${workspaceFolder}/blinky/build/zephyr/zephyr.elf", "description": "load file", "ignoreFailures": false},
{ "text": "-file-exec-and-symbols C:\\git\\nrf_blinky_stlink\\blinky\\build\\zephyr\\zephyr.elf", "description": "load file",

As you can see separators need to be escaped twice in order to work. Once for json and once for the command line application. So even pathSeparator doesn't work unless you use it twice. Enclosing it in escaped double quotes doesn't solve the problem.
Steps to Reproduce:

Use a path that that doesn't work
Result: command line program(gdb in my case) responds with "No such file or directory"

I've noticed there are similar issues raised, but some of them are slightly different (they apply to tasks, not launch, for instance). I haven't seen a solution yet (other than using git bash). So I propose the following solution:
Create variants of workspaceFolder so you can convert the path of your workspace folder to have the separators you want, regardless of your operating system.
For example: workspaceFolderEscaped; same as workspaceFolder, but with \ replaced with \\ or with /, on Linux it has no effect so it can also be used on Linux in theory

What are you trying to do with the "command" build like that?

I have an example here where I need to pass a path to an elf file in gdb: https://github.com/ChrisIdema/nrf_blinky_stlink/ .  I've run into similar issues before, but this is the most recent example.

It would help if you could provide the full json sample (e.g. the launch.json or tasks.json).
I cannot investigate the problem if you only show something like this:
{ "text": "-file-exec-and-symbols C:\\\\git\\\\nrf_blinky_stlink\\\\blinky\\\\build\\\\zephy...

It would help if you could provide the full json sample (e.g. the launch.json or tasks.json).

https://github.com/ChrisIdema/nrf_blinky_stlink/
https://github.com/ChrisIdema/nrf_blinky_stlink/blob/master/.vscode/launch.json
https://github.com/ChrisIdema/nrf_blinky_stlink/blob/master/.vscode/tasks.json

@ChrisIdema Thanks for the links.
I do not know how the entries in the setupCommands array are used, but I'm sure that the code that uses them has to deal with with escaping special characters. This is not VS Code but the cpp extension.

/extCpp

It looks like this is caused by the C++ extension. Please file the issue to the C++ extension repository. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!

I've submitted a new issue for the debug part to the C++ extension (https://github.com/microsoft/vscode-cpptools/issues/9785).  But you missed the part where I said the tasks.json has the same issue. So please reopen as the C++ extension is not responsible for this task:
        {
            "label": "flash",
            "group": "build",
            "type": "shell",
            "windows":{
                "command": "openocd.exe",
                "args":[
                    "-d2",
                    "-f",
                    "openocd.cfg",
                    "-c",
                    "'init_reset halt; program \"C:/git/nrf_blinky_stlink/blinky/build/zephyr/zephyr.hex\" verify; reset; exit'",
                    // "'init_reset halt; program \"${workspaceFolder}/blinky/build/zephyr/zephyr.hex\" verify; reset; exit'", // doesn't work
                ],
            },
            "linux":{
                "command": "openocd",
                "args":[
                    "-d2",
                    "-f",
                    "openocd.cfg",
                    "-c",
                    "'init_reset halt; program \"${workspaceFolder}/blinky/build/zephyr/zephyr.hex\" verify; reset; exit'",
                ],
            },
            "problemMatcher": [],
        },

Like I said workspaceFolderEscaped would be a perfect candidate here. Requiring absolute path literals in configurations is simply not proper as it is depends on where you clone the repo.

@alexr00 do you have suggestions for how to solve the tasks.json related issue in the previous comment?

I'm currently experimenting with using environment variables assigned prior to running the command in the same task. It seems like it's not working with nested commands (openocd calls gdb).
This is what I have now:
{
    "label": "make wsf",
    "group": "build",
    "type": "shell",
    "windows":{
        "command": "\"$Env:wsf='${workspaceFolder}'; $Env:wsf=$Env:wsf -replace \\\"\\\\\\\\\\\",\\\"/\\\"; echo $Env:wsf\"",
    },
    "problemMatcher": [],
},

Which simply echos the converted path: C:/git/nrf_blinky_stlink
But I want to use it with openocd. So I wrote this:
{
    "label": "flash2",
    "group": "build",
    "type": "shell",
    "windows":{
        "command": "\"$Env:wsf='${workspaceFolder}'; $Env:wsf=$Env:wsf -replace \\\"\\\\\\\\\\\",\\\"/\\\"; echo $Env:wsf; openocd -d2 -f openocd.cfg -c 'init_reset halt; program $Env:wsf/blinky/build/zephyr/zephyr.hex verify; reset; exit'\"",
    },
    "problemMatcher": [],
},

But then I get this error:
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
can't read "Env": no such variable

It seems like I'm almost there. For some reason I cannot pass the environment variable to openocd, but I can pass it to echo. But if I get it to work it won't be an elegant solution. I could write a batch script, but in my opinion that should not be required.

@weinand I think this should this issue be reopened. Our debugger developer says "As it is also an issue in the task.json escaping, this is pointing at a VS Code variable resolution issue. VS Code resolves the variables in resolveDebugConfigurationWithSubstitutedVariables".

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

:slightly_frowning_face: In the last 60 days, this feature request has received less than 20 community upvotes and we closed it. Still a big Thank You to you for taking the time to create this issue! To learn more about how we handle feature requests, please see our documentation.
Happy Coding!


STC method: algorithm?

Dear Dr. Thomas,
Using different reference regions to intensity normalize my PET images (cerebellum, pons,...), the mean PVC of each reference region is not always available based on the PVC method used.
I was wondering whether STC would be a good alternative to determine the mean PVC of all my reference regions? Which algorithm is based STC on?
Best regards,
Matthieu

This would be interesting to try. STC is conceptually similar to multi-target correction (MTC) (see Erlandsson et al., Phys. Med. Biol. 57 (2012) R119–R159) except the correction is calculated on a voxel-wise basis rather than using regional means, as in MTC.


MySQL: only serialize non-sensitive data in db.statement attribute

Is your feature request related to a problem? Please describe
When I use mysql instrumentation, I see the full query in the span: db.statement: SELECT a FROM b WHERE password='ppp'.
I think we shouldn't display sensitive data by default.
Describe the solution you'd like to see
In mongo/mongoose, and in redis/ioredis we have a config object that has a field called: dbStatementSerializer which is a function that determines how the statement will be parsed. I think we need to add this ability to mysql also.
Describe alternatives you've considered
I thought to add a boolean parameter that enables/disables the db.statement.
But I think this solution is too "narrow". With a function, the user can decide how the db.statement will look like and not just decide if to present it or not.

If accepted, can you assign this to me?

Thanks for opening the issue. Is it the same as #1237 ?

duplicate of #1237

duplicate of #1237


Added an --all-versions flag to gen-api-docs and clean-api-docs

Description
Added an --all-versions flag to gen-api-docs and clean-api-docs in order to be able to generate all versions of all apis without having to individually specify each api you would like to generate all versions of.
Motivation and Context
I am trying to build a docker image of docusaurus + openapi-docs into which I can mount any arbitrary docs folder + api specs and it will detect all the apis present and generate docs for them. I do this using a function in docusaurus.config.ts. When starting the docker container it runs docusaurus gen-api-docs all to generate the docs. Unfortunately this only generates the current version of each api and not the historical versions. I couldn't think of a way to make the list of apis available to the docusaurus gen... command, so instead I'm proposing this PR, which adds a -all-versions flag to docusaurus gen-api-docs and docusaurus clean-api-docs to support this use case.
I think it's also a QOL improvement as I find having to generate versions separately from the docs is a bit aggravating.
How Has This Been Tested?
I ran tests. It passes.
I ran tests:cypress. It doesn't pass, but it didn't pass for me on main either.
I replaced the gen-all and clean-all scripts which generate all docs and then specifically generate all versions for petstore_versioned with this flag and compared the outputted files from the two using mtree. There were no differences.
I'm not much a javascript developer, so I wasn't sure what more I could/should be doing. Happy to receive feedback/suggestions.
I don't believe my change will impact any other parts of the code.
Screenshots (if appropriate)
n/a
Types of changes

New feature (non-breaking change which adds functionality)

Checklist

[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes if appropriate.
[ ] All new and existing tests passed.
I couldn't get existing tests to pass on main and I'm not sure what other tests would be appropriate as I couldn't find any existing tests for the cli.

Hi @dv-stewarts, thanks for the contribution. I haven't had time to test but I think the approach should work. Will try to take a look either this or early next week. Thanks!

Hi @dv-stewarts, while I was reviewing your branch I noticed something: the running yarn docusaurus --help no longer includes the extended commands introduced by docusaurus-plugin-openapi-docs. I don't think it's anything specific to your changes but definitely changed between v2 and v3. I checked the most recent docs on extendCli and didn't see anything different about the usage.
For example:
Before (v2)
$ docusaurus -h
Usage: docusaurus <command> [options]

Options:
  -V, --version                                            output the version number
  -h, --help                                               display help for command

Commands:
  build [options] [siteDir]                                Build website.
  swizzle [options] [themeName] [componentName] [siteDir]  Wraps or ejects the original theme files into website folder for customization.
  deploy [options] [siteDir]                               Deploy website to GitHub pages.
  start [options] [siteDir]                                Start the development server.
  serve [options] [siteDir]                                Serve website locally.
  clear [siteDir]                                          Remove build artifacts.
  write-translations [options] [siteDir]                   Extract required translations of your site.
  write-heading-ids [options] [siteDir] [files...]         Generate heading ids in Markdown content.
  docs:version <version>                                   Tag a new docs version
  gen-api-docs [options] <id>                              Generates OpenAPI docs in MDX file format and sidebar.js (if enabled).
  gen-api-docs:version [options] <id:version>              Generates versioned OpenAPI docs in MDX file format, versions.js and sidebar.js (if enabled).
  clean-api-docs [options] <id>                            Clears the generated OpenAPI docs MDX files and sidebar.js (if enabled).
  clean-api-docs:version [options] <id:version>            Clears the versioned, generated OpenAPI docs MDX files, versions.json and sidebar.js (if enabled).

Current (v4)
$ docusaurus -h
Usage: docusaurus <command> [options]

Options:
  -V, --version                                            output the version number
  -h, --help                                               display help for command

Commands:
  build [options] [siteDir]                                Build website.
  swizzle [options] [themeName] [componentName] [siteDir]  Wraps or ejects the original theme files into website folder for customization.
  deploy [options] [siteDir]                               Deploy website to GitHub pages.
  start [options] [siteDir]                                Start the development server.
  serve [options] [siteDir]                                Serve website locally.
  clear [siteDir]                                          Remove build artifacts.
  write-translations [options] [siteDir]                   Extract required translations of your site.
  write-heading-ids [options] [siteDir] [files...]         Generate heading ids in Markdown content.

Update: I went ahead and opened https://github.com/facebook/docusaurus/issues/10681

Thanks @sserrata. I doubt my change could have broken that, but since I, at best, dabble with javascript, anything is possible.

@sserrata I see docusaurus have closed https://github.com/facebook/docusaurus/issues/10681 Do you think this can be merged in?

Hey @dv-stewarts, yes I saw that. I suppose we could use the latest canary release from Docusaurus to test - just want to make sure the help output/usage looks good before proceeding. I suppose we could also just pass a bogus command since the error output appears to include the extended commands.

Ah, since it's a command option, and not a command, it does show after all:
 docusaurus-openapi-docs/demo  all_versions_flag yarn docusaurus clean-api-docs -h
yarn run v1.22.19
$ docusaurus clean-api-docs -h
Usage: docusaurus clean-api-docs <id>

Clears the generated OpenAPI docs MDX files and sidebar.ts (if enabled).

Options:
  -p, --plugin-id <plugin>  OpenAPI docs plugin ID.
  --all-versions            Clean all versions.
  -h, --help                display help for command
✨  Done in 1.99s.

 docusaurus-openapi-docs/demo  all_versions_flag yarn gen-api-docs -h
yarn run v1.22.19
$ docusaurus gen-api-docs -h
Usage: docusaurus gen-api-docs <id>

Generates OpenAPI docs in MDX file format and sidebar.ts (if enabled).

Options:
  -p, --plugin-id <plugin>  OpenAPI docs plugin ID.
  --all-versions            Generate all versions.
  -h, --help                display help for command
✨  Done in 2.23s.

Question...just to clarify behavior, the idea is that --all-versions will work with either all or a key referencing a versioned API, e.g. petstore_versioned, but NOT a non-versioned key, e.g. petstore. Is that correct?

@sserrata I think the intention was to generate all versions, regardless of if versions exist or not. If there is only one version, then generating non-versioned docs seems to me like the behavior that was requested.
Either way, thanks for merging!


Fix performance regression in queue

Addresses #839
This attempts to fix a performance regression in #775 by removing max_tries logic from queue and using cheaper max_tries logic in Sample (the only place max_tries is be used).

This logic is incorrect.  max_tries is the maximum number of times to try returning a single value from queue, not the number of times to invoke it.

Oops sorry didn't mean to close

@eb8680 Can you create a simpler PR that fixes this by
from six.moves import xrange

for i in xrange(max_tries):
   ...


Container image is missing 'unstructured' pip package

Resulting in these errors:
Setting up quick upload event
Running on local URL:  http://<IP_ADDRESS>:7860

To create a public link, set `share=True` in `launch()`.
User "admin" already exists
Setting up quick upload event
User-id: None, can see public conversations: False
User-id: 1, can see public conversations: True
len(results)=0, len(file_list)=1
len(results)=0, len(file_list)=1
Overriding with default loaders
use_quick_index_mode False
reader_mode default
Using reader <kotaemon.loaders.unstructured_loader.UnstructuredReader object at 0x7f8d1a655e10>
use_quick_index_mode True
reader_mode default
Using reader <kotaemon.loaders.unstructured_loader.UnstructuredReader object at 0x7f8d1a655e10>
No module named 'unstructured'
Traceback (most recent call last):
  File "/app/libs/ktem/ktem/index/file/pipelines.py", line 724, in stream
    file_id, docs = yield from pipeline.stream(
  File "/app/libs/ktem/ktem/index/file/pipelines.py", line 586, in stream
    docs = self.loader.load_data(file_path, extra_info=extra_info)
  File "/app/libs/kotaemon/kotaemon/loaders/unstructured_loader.py", line 70, in load_data
    from unstructured.partition.auto import partition
ModuleNotFoundError: No module named 'unstructured'
/usr/local/lib/python3.10/site-packages/gradio/components/dropdown.py:188: UserWarning:

The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: None or set allow_custom_value=True.

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."

That's obviously a virus/scam - don't click that if you're reading this ^

Thanks for the quick feedback. Default Docker image is missing a lot of file loaders required package like the one you mentioned above. We will provide more feature-rich Docker image in the future with dependencies pre-installed (but image size will increase dramatically).

Same issue for me

After downloading 0.4.1 release same error happens for me using "packaged" version. Im just trying to index .txt file
len(results)=1, len(file_list)=1
use_quick_index_mode False
reader_mode default
Using reader <kotaemon.loaders.unstructured_loader.UnstructuredReader object at 0x000001C8375C7AC0>
No module named 'unstructured'
Traceback (most recent call last):
  File "D:\software\kotaemon-app\install_dir\env\lib\site-packages\ktem\index\file\pipelines.py", line 724, in stream
    file_id, docs = yield from pipeline.stream(
  File "D:\software\kotaemon-app\install_dir\env\lib\site-packages\ktem\index\file\pipelines.py", line 586, in stream
    docs = self.loader.load_data(file_path, extra_info=extra_info)
  File "D:\software\kotaemon-app\install_dir\env\lib\site-packages\kotaemon\loaders\unstructured_loader.py", line 70, in load_data
    from unstructured.partition.auto import partition
ModuleNotFoundError: No module named 'unstructured'
len(results)=1, len(file_list)=1

pip install unstructured

solved it for me

Resolved in the new version. Please check the latest docker full


allow string device input for OrtModel.to(device)

What does this PR do?
Implements #336

@jessecambon You need to rebase your branch on the main branch so that the tests pass

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@regisss thanks, I also added the same modification to the sequence to sequence model and added tests. My fork should now be up-to-date with main.

@regisss thanks, I also added the same modification to the sequence to sequence model and added tests. My fork should now be up-to-date with main.

Thanks a lot @jessecambon :smiley:
It all looks good to me. I just launched the tests and if they all pass I will merge this PR.


ban bypass

Add a way for servers to bypass the bans and on the game so mojang can’t do anything and hide the warnings or allow people to report anyways so mojang gets a ton of banned people reported

i don't think thats is possible because mojang well simply tells the server thats the account is unauthenticated like a cracked game

hide the warnings is something that can be done and has already been done

Ok

Make the server check for can and override with the offline mode for that player

There is no simple way to implement this, and less simple ways are out of scope for this project. I would want it to remain focused on providing preventive measures.


Add custom model and max tokens options in settings

https://github.com/HyeonseoNam/auto-classifier/issues/10

@HyeonseoNam
Hi. Could you review this PR to merge? It's working fine for me.

Hi @tk42,
Thank you for your effort, this looks good !


Github icon

The Github icon on https://upbge.org/#/community looks really bad. Can you replace it with SVG?
(You can get it right here https://github.com/logos)

Agree


Remove previous instance of trained model when re-training

I discovered today that when you train a model that has been already trained with a different input datasets (which may have a different energy binning) the previous instance of the trained model is not deleted. This gives errors in the later stages of the analysis (e.g. will extract all energy bins, from both instances).

Yes, I am aware of this and it annoyed me to no end until now. Never got around to fixing it =)
Will try to do it soon.

Hi Orel
No rush at all! I just stumbled with this problem and lost some time until I realized about it... If you are aware of it, then another thing in our to-do list...


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_e5h2fkc0if/clone/recipe/meta.yaml:15: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_e5h2fkc0if/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_e5h2fkc0if/clone/recipe/meta.yaml:30: missing_pip_check: For pypi packages, pip check should be present in the test commands
WARNING: /tmp/abs_e5h2fkc0if/clone/recipe/meta.yaml:39: missing_description: The recipe is missing a description
Errors were found


_TZE200_uiyqstza does not work

Hi! 👋
Since last year I've got two _TZE200_i48qyn9s and they're working fine. I thought I'd buy more of those, but the new ones that arrived today are _TZE200_uiyqstza and they don't work with the same quirk.
I admit that I didn't yet have the time to try to fix the problem myself, but I think at least I got you the logs of when I pair the device.
_TZE200_uiyqstza.txt
I think this is similar to https://github.com/jacekk015/zha_quirks/issues/11
Thanks for everything and I'll try to help with this as good as I can.

Did you downloaded and used latest version of quirk??
ts0601_trv_maxsmart.py

I began looking into it a bit. Remember that I have no idea what I'm doing, sorry. Maybe I'm just wasting time 🤷
The endpoint information of my device does indeed not match the Maxsmart definition: endpoint=1, profile=260, device_type=81, device_version=1, input_clusters=[4, 5, 61184, 0], output_clusters=[25, 10].
It looks very similar to the Saswell quirk, so I just went ahead and added _TZE200_uiyqstza to the saswell MODEL_INFO.
That doesn't work either though. Trying more stuff.

The device signature is from the Zigbee module only the manufacture and model its getting from the MCU and so if the manufacture is changing the Zigbee module but have the same firmware in the MCU must one new device class being made and for matching the signature but the rest  shall being the same.
It was the same with first gen devices was having different Zigbee modules but the same MCU.

@Kampfgnom Redownload newest ts0601_trv_maxsmart.py quirk from my repo.
Maybe it will not work properly, but it should start to talk now with HA by Zigbee.
Restart HA after quirk replace.

Wait 10 minutes minimum and post logs again.

I fixed the one obvious thing: DEVICE_TYPE: zha.DeviceType.SMART_PLUG.
The device now pairs, but does not finish and is not yet usable. Here are logs:
_TZE200_uiyqstza-20221026.txt

It keeps asking for time request. Probably special initialization is needed.
Try updated quirk.

That's a step forward, thanks!
The device is now configured and it reads the initial values, partially at least.
It doesn't receive updated values and it cannot set values though.
_TZE200_uiyqstza-20221026T1350.txt

Re-download and try.
Still got set_time with errors from TRV.
I've made an exception for that TRV which normally worked - but I've never had problems with set_time command.
Try, post the logs, and drop a screenshot from device if possible.
On Win10 you can do shortcut Shift + Win + S - mark the area and click on Github post window and do Ctrl+v [paste].
I will paste and upload automagically.

Now the configuration itself does not finish and the device does not have entities at first.
After a HA restart it has entities. Some entities show correct values (e.g. 17°C), other's don't. The thermostat instance does not work. Setting values does not work.
I collected logs when I added the device and also for after the restart:
_TZE200_uiyqstza-20231030T1738.txt
_TZE200_uiyqstza-20231030T1738_after_restart.txt
Here's an iOS screenshot, after the restart, because it includes the values (why aren't they shown in the browser?):

.txt)
I finally have some time to look into it myself. I'll play around a bit. If you have any pointers as to what to try, maybe we can reduce the feedback cycle times because I manage to get something done on my own?
Thanks for everything so far 👍

After replacing quirk file you ALWAYS need to do restart. And then re-pair.
Quirk code is ONLY loaded at the HA start/restart.

Sorry! That's what happens when you mess around with this stuff while your toddler is also trying to get your attention. I had installed a version where I changed some things myself.
Now I'm running your https://github.com/jacekk015/zha_quirks/commit/3931d162043991acef484f623e5f8db876c850a8 and most things already work 👍
Pairing:
_TZE200_uiyqstza-20231031T1530_pairing.txt
Setting temperature from HA UI, does not work:
_TZE200_uiyqstza-20231031T1530_set_temperature.txt
Changing the temperature on the TRV, reports back to HA successfully:
_TZE200_uiyqstza-20231031T1530_trv_change_temperature.txt
iOS UI:

I begin to understand some things. Setting is_manufacturer_specific=False makes setting the temperature work 🙂
        0x0000: foundation.ZCLCommandDef(
            "set_data",
            {"param": TuyaManufCluster.Command},
            False,
            is_manufacturer_specific=False,
        ),

I played around with the TRV a bit and now everything seems to work as expected. I will test-drive it for a while and report back whether now everything is working.

Do you accept donations somehow? Or would you like to suggest an organization or individual you would like me to donate to in your place? :)

begin to understand some things. Setting is_manufacturer_specific=False makes setting the temperature work 🙂

Yes and no. Chinese friends probably don't like standards.
On pairing your device presented manufacturer ID=4417
Oct 31 14:23:26 homeassistant homeassistant[576]: 2023-10-31 15:23:26.930 INFO (MainThread) [zigpy.device] [0x37dd] Got Node Descriptor: NodeDescriptor(logical_type=<LogicalType.EndDevice: 2>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress: 128>, manufacturer_code=4417, maximum_buffer_size=66, maximum_incoming_transfer_size=66, server_mask=10752, maximum_outgoing_transfer_size=66, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)

So ZHA/HA remember's that and on every next command send back to TRV uses that manufacturer ID. Why not?
That what normally is_manufacturer_specific=True does - it's a standard.
But if that works that way then some TRV's, for different commands, barks back with:
UNSUP_MANUF_CLUSTER_COMMAND
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.632 DEBUG (MainThread) [zhaquirks.tuya] [0x37dd:1:0x0201] Mapping standard occupied_heating_setpoint (0x0012) with value 2000 to custom {617: 40}
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.633 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Sending request header: ZCLHeader(frame_control=FrameControl(frame_type=<FrameType.CLUSTER_COMMAND: 1>, is_manufacturer_specific=True, direction=<Direction.Server_to_Client: 0>, disable_default_response=0, reserved=0, *is_cluster=True, *is_general=False), manufacturer=4417, tsn=76, command_id=0, *direction=<Direction.Server_to_Client: 0>)
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.634 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Sending request: set_data(param=Command(status=0, tsn=76, command_id=617, function=0, data=40))
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.814 DEBUG (MainThread) [zigpy.application] Received a packet: ZigbeePacket(src=AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0x37DD), src_ep=1, dst=AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0x0000), dst_ep=1, source_route=None, extended_timeout=False, tsn=0, profile_id=260, cluster_id=61184, data=Serialized[b'\x18L\x0b\x00\x83'], tx_options=<TransmitOptions.NONE: 0>, radius=28, non_member_radius=0, lqi=36, rssi=None)
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.815 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Received ZCL frame: b'\x18L\x0b\x00\x83'
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.816 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Decoded ZCL frame header: ZCLHeader(frame_control=FrameControl(frame_type=<FrameType.GLOBAL_COMMAND: 0>, is_manufacturer_specific=0, direction=<Direction.Client_to_Server: 1>, disable_default_response=1, reserved=0, *is_cluster=False, *is_general=True), tsn=76, command_id=11, *direction=<Direction.Client_to_Server: 1>)
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.817 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Decoded ZCL frame: Silvercrest3ManufCluster:Default_Response(command_id=0, status=<Status.UNSUP_MANUF_CLUSTER_COMMAND: 131>)
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.818 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Received command 0x0B (TSN 76): Default_Response(command_id=0, status=<Status.UNSUP_MANUF_CLUSTER_COMMAND: 131>)

That's because if you send data to TRV, like request to change Temp, you send a manufacturer ID, which that TRV presented to you on the pairing process.
But now Chinese firmware in the TRV says nope - I don't like that manufacturer ID. Below you have manufacturer=4417
Oct 31 14:25:32 homeassistant homeassistant[576]: 2023-10-31 15:25:32.633 DEBUG (MainThread) [zigpy.zcl] [0x37DD:1:0xef00] Sending request header: ZCLHeader(frame_control=FrameControl(frame_type=<FrameType.CLUSTER_COMMAND: 1>, is_manufacturer_specific=True, direction=<Direction.Server_to_Client: 0>, disable_default_response=0, reserved=0, *is_cluster=True, *is_general=False), manufacturer=4417, tsn=76, command_id=0, *direction=<Direction.Server_to_Client: 0>)

That's why we override those commands and do then:
is_manufacturer_specific=False
Probably the reason is that those are clones of the clones and no one care about that. They're Zigbee gateways probably totally omit anything with manufacturer ID so it just works for them.

My four _TZE200_uiyqstza have been running nicely since last week.
For future reference, they're called "Wesmartify Essentials Heizkörperthermostat Zigbee (120144)" and I got them from here:
https://www.pollin.de/p/essentials-heizkoerper-thermostat-zigbee-590612?etcc_med=pvgl&etcc_ori=idealo.de&et_cmp_seg2=pvgl&channelid=psm&utm_medium=pvgl&utm_source=idealo.de
via
https://www.idealo.de/preisvergleich/OffersOfProduct/200684364_-essentials-heizkoerperthermostat-zigbee-120144-wesmartify.html
All the features I need are in place 👍
I'm running https://github.com/KartoffelToby/better_thermostat
I have several automations which set the target temperature.
It might not be fully feature complete, but my use case is fulfilled.
Thanks again for everything 🙂


How to run pretrained resnet50 from the model zoo

Hi all,
I am a bit new to caffe2. I followed this python tutorial on running pretrained models and can make it work with alexnet, squeezenet, etc.:
https://github.com/caffe2/caffe2/blob/master/caffe2/python/tutorials/Loading_Pretrained_Models.ipynb
But it does not work with the pretrained resnet50 model here:
https://github.com/caffe2/models/tree/master/resnet50

I looked at its predict_net.pbtxt (below) which is different from alexnet's:
op {
input: "gpu_0/data"
input: "gpu_0/conv1_w"
output: "gpu_0/conv1"
name: "
type: "Conv"
arg {
...
If I just change {"data":img} to {"gpu_0/data":img}, another error pops up:

Can anyone suggest the right way to run this model? Thanks a lot in advance.
-- J

Has your input image the correct dimensions? Here you can find the specification:
https://github.com/caffe2/models/blob/master/resnet50/value_info.json
I can reproduce your error when I set my image dimension to 227x227 pixels. When I change the crop dimension to 224x224 resnet50 works without errors.
Can you try to change the dimension (after cropping) to be 224x224 pixels and 3 channels?

@SteveCruz Absolutely you are right. Thanks a lot.
-- J

Thanks @SteveCruz for the assist on this one. Closing.

@SteveCruz  @orionr
I can get the right prediction using the squeezenet, but when i deploy resnet50 to predict the label of some images, all of are predicted incorrectly as spotlight, could you please tell me the reason?
BTW, I use  224x224 pixels and 3 channels and change {"data":img} to {"gpu_0/data":img}.


Getting a 100% offline IPT to load core files + extensions in admin/extensions

Hiya, I'm in Armenia doing a BioDATA training course with @laurarussell. We've got a patchy internet connection, so I'm trying to set up a new version of the IPT on our raspberry pi for our students to connect to.
We've unfortunately got no way of connecting the pi to the internet, so I've downloaded the latest IPT war, then put that on the pi where we have tomcat8, and it's deployed it. I added our students as users and everything is looking fine, but I don't seem to be able to trick it into loading the core files so that our students can actually use it. I copied over the .extensions and .vocabularies directories from the config folder of another online ipt (of the same version) and put them on the pi, but no dice, the error remains -There are no Core Types installed. I don't think there's a file permissions problem, I've fiddled with that a bit and still not got it working. Needless to say, I have restarted tomcat maaaaaany times.
Anyway, I'm certain that I've done this before previously with this same raspberry pi in pre-covid times, but I don't remember how I got it working. @laurarussell suggested I drop you guys a line and ask for a bit of help :) @MattBlissett any ideas?

Hi,
We haven't tested or planned for a completely offline IPT.
I think you'll need to make a fake gbrds.gbif-uat.org (registry) and possibly rs.gbif.org (schemas and extensions), for example using Apache installed on the Pi.
mkdir -p gbrds.gbif-uat.org/registry/organisation
cd gbrds.gbif-uat.org/registry
for i in extensions.json network.json organisation.json thesauri.json; do wget http://gbrds.gbif-uat.org/registry/$i; done
touch organisation/625a5522-1886-4998-be46-52c66dd566c9.json

wget -r -R '*C=*' -R index.html http://rs.gbif.org

<VirtualHost *:80>
        ServerName rs.gbif.org
        DocumentRoot /home/matt/offline-ipt/rs.gbif.org

        <Directory /home/matt/offline-ipt/rs.gbif.org>
                   Require all granted
        </Directory>
</VirtualHost>

<VirtualHost *:80>
        ServerName gbrds.gbif-uat.org
        DocumentRoot /home/matt/offline-ipt/gbrds.gbif-uat.org

        <Directory /home/matt/offline-ipt/gbrds.gbif-uat.org>
                   Require all granted
        </Directory>
</VirtualHost>

Thanks @MattBlissett! I tried this with little success, and then we found a way to open up the pi to the internet. And now I have a new error :D
 An IO error occurred while trying to communicate with the GBIF Registry. [https://gbrds.gbif-uat.org/registry/thesauri.json]
The server can certainly access https://gbrds.gbif-uat.org/registry/thesauri.json as far as I can tell. I see that it's probably coming up here: https://github.com/gbif/ipt/blob/98c745a7e9e3eeb9f779420ef277ad658ea20467/src/main/java/org/gbif/ipt/service/registry/impl/RegistryManagerImpl.java#L470
I guess there is probably some file permissions problem, I suppose the IPT is trying to download thesauri.json to a temp folder somewhere and it's not been able to. To fix it I've tried reinstalling tomcat and setting 777 permissions on absolutely everything tomcat/ipt related that I come across in the filesystem. But it's still complaining.
This is kind of an academic problem now, the students will just have to (as well as they can with the internet problems) use the online IPT we set up for them on the server in Oslo (the session is up next in 15 minutes).  I'm still sort of interested in finding out what's going wrong though. It's probably something really simple which I should have set up in the tomcat installation that I missed.

Is the Pi using a Debian/Ubuntu derivative?  It could be this: https://ipt.gbif.org/manual/en/ipt/2.5/faq#sandboxing

It is! I will take a look, thanks :)

Oh dear... It doesn't seem to be that. We're going to nuke it and start over from scratch. Will let you know how it goes! Thanks for your help though, much appreciated :)

Is this issue still relevant please?

No, it can be closed.


However, this is not something we are interested in adding.
Next time please create a discussion first to propose the change.


Value::tryGetBoolean()  does not return defaultValue as documented

The documentation indicates that Value::tryGetBoolean() returns Value's boolean value, or defaultValue if the value doesn't contain a boolean.  However, if the value doesn't contain a boolean it actually returns EMPTY_BOOL instead of returning defaultValue.

Thank you for finding this bug, I have fixed it.


Cypress 4.x support

Does anyone know when Cypress 4.x might be supported?
We have an application that needs the extra functionality provided in 4.x.
I already verified with our development team, and 3.x does not have everything that we need.
Thanks in advance, and have a great week.
JCF

Hello @jxfish2,
Cypress 4.0.0 was released on July 26, 2018. It includes:

Support for the 2018 CQL-based Electronic Clinical Quality Measures for 2019 Calendar Year reporting
Support for the CMS QRDA Cat I Implementation Guide for the 2019 CY Reporting Period
Support for the HL7 QRDA Cat I R5 and QRDA Cat 3 R2.1 Implementaton Guides

Information on how to download and install Cypress 4.0.0 can be found on our wiki here.
Please note that we anticipate the release of a Cypress v4.0.1 patch release on October 11, 2018.
If you have any additional questions about Cypress, please feel free to ask us, or to post an issue at https://oncprojectracking.healthit.gov/support/browse/CYPRESS
Thank you,
Michael O'Keefe

Hi Michael,
Thank you for the response.
Unfortunately, the reason for my post here, was because I had already been to that page, and as I understand it, the new Cypress 4.x is not supported in the format that we need.
My company decided to migrate everything we have to the Azure cloud, instead of to the Amazon cloud.
It appears as if the current image that you offer is in a "VHD" format, but for the Azure cloud, it needs to be in a "VMDK" format.
The "VHD" image will not work in Azure.
Now, Azure does offer a pre-built and configured Ubuntu image with Docker already installed.
If 4.0 has a Docker image that can be downloaded to an Ubuntu server already running Docker, that might work.
Or, if 4.0 has a Docker image that can be downloaded to a CentOS server already running Docker, that would also work.
Note that we use CentOS and RHEL in house, and at customer sites.  We only use Ubuntu, when an application demands Ubuntu.
Any ideas?
Thanks in advance, and have a great day.
JCF

Hello @jxfish2,
Thanks for the response! I'll re-open this ticket, and discuss with the other DevOps people on the Cypress team.

It should be very easy to deploy Cypress using our chef scripts, the instructions for how to do so can be found here: https://github.com/projectcypress/cypress-recipe#chef-solo
If that doesn't work or you have any issues please feel free to let me know.

Ok, I will try.
Thanks for your time, and have a great afternoon.
JCF

@jxfish2 I'm going to close this ticket as resolved, given the suggestion to use the Chef install script. If you continue to have issues, feel free to comment/re-open the ticket.
Thank you,
Michael


In AppRevision components have no name

Currently in AppRevision the components have no name like below:
  applicationConfiguration:
    apiVersion: core.oam.dev/v1alpha2
    kind: ApplicationConfiguration
    metadata: {}
    spec:
      components:
      - revisionName: testsvc-v1
    status:
      dependency: {}
      observedGeneration: 0
  components:
  - apiVersion: core.oam.dev/v1alpha2
    kind: Component
    metadata: {}
    spec:
...

The problem is that there is no way to correlate the  components and ACComponents.

This is due to applying another k8s object in an object.
type ApplicationRevisionSpec struct {
	// Components records the rendered components from Application, it will contains the whole K8s CR of workload in it.
	Components []*Component `json:"components"`

	// ApplicationConfiguration records the rendered applicationConfiguration from Application,
	// it will contains the whole K8s CR of trait and the reference component in it.
	ApplicationConfiguration ApplicationConfiguration `json:"applicationConfiguration"`

We should change to type runtime.RawExtension.


Gitlab support

Will SPR be a Github only tool? Or, in the current developers' opinions, is SPR well suited to be extended for other code forges, like Gitlab?

Currently spr only works with the GitHub api.
It looks like GitLab also has a graphql api, the api interface is mocked out in the code, so it should be possible to add a GitLab implementation without needed to change the core application.
Someone using GitLab would have to implement this interface in order to add support for GitLab.

Just what I wanted to know! Thanks.


sent bulk size of 100 fwjr report to WMArchive and set the polling cycle to 10 min.

fixed in #6719


add school

@Bchaises Please provide us with:

the school official website URL
the school street address, including city and country
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

@Bchaises You might miss my previous comment. Please react.

@Bchaises I am sorry, this request remains unanswered for a while. I have to close it without merging. Please reopen and comment if you have an update regarding the aforementioned issue. Thank you for your understanding!


Fix usermod error UID 0 already exists.

This bug prevented rancher/server from starting when a non-vboxfs
directory was used as the mount point for /var/lib/mysql. In the
case of docker-machine/boot2docker any directory outside of /Users
on a Mac or non rancher/server container volumes from meet this
criteria. VBoxFS mounts do not allow changing UID/GID or permissions
of the directory. To address we change the UID of the MySQL user
to the directory owner UID.

Addresses #1972


Fixed issue with WebSocket measurement stream non-default timeout bei…

Timeout value was being treated incorrectly.

Verified fix


Define warning() when in debug mode

I noticed that multiple places in the code did had the warning function
commented out, so instead of just removing them the better solution
was to use the already existing release/debug system. When compiling
under debug mode, CMake defines DEBUG=1 which in turn enables the
warning() function. Otherwise it's an empty macro.

The warning() function is only compiled under debug mode. Using errno would surely provide more insight about whats going on to the user, which is most of the time a good thing.

please make your pull request to develop branch

good one


Format using directives

Fixes https://github.com/dotnet/aspnetcore/issues/34961
Intended to work on https://github.com/dotnet/aspnetcore/issues/32419
Found nothing wrong, decided to format using statements anyway.
I think I logged that issue based on an assumption of razor syntax that doesn't hold true.

/azp run


Add partition key

@dasch https://github.com/zendesk/racecar/pull/169/commits/461fab0a0765fc62573a496e0f566f3581850452 fixed the failures for me locally.

Thanks!


Informal proposal for strong mode toplevel inference

This is an informal specification of proposed changes to strong mode top level inference (https://github.com/dart-lang/sdk/issues/27499).

@leafpetersen can we land this?

@leafpetersen can we merge this?

We shouldn't merge it.  I'd rather move this to the language repo.  I'll sort it out next week and then close this.


Instantiate execution plan capture callback via shim loader

Closes https://github.com/NVIDIA/spark-rapids/issues/9103
This PR puts ExecutionPlanCaptureCallback under the shims so it can use shimmed classes such as GpuFileSourceScanExec. I was able to repro the failure seen in the MT CI job locally and this approach fixes it.

build

build

build

build

This should be ready to go in. If folks here don't mind to take a look @jlowe @gerashegalov @andygrove


Add left arrow <- token

The left arrow symbol has been unused since before Rust 1.0, but it is still treated as a single token

https://doc.rust-lang.org/reference/tokens.html#punctuation

I noticed while using macros that call for the <- token that it gets highlighted as < and - separately. rustc parses it as a single token, as demonstrated here.

Thanks!


Create Device Models Repository Client

This includes the untested new code structure for the Device Models Repository Client, following closely to the architecture of the Python Models Repository Client (https://github.com/Azure/azure-sdk-for-python/pull/17180)
It is currently untested, and likely has bugs. In addition the tests for Node.js are not complete and neither is the rollup configuration. Futher testing using karma must be conducted to validate browser compatibility, based on shimming the filesystem calls and other node specific utilities.

@chradek can you take a first pass through this?

Note: Though the node tests are working, we haven't figured out browser tests yet. Those are failing.

/azp run prepare-pipelines


How to run cargo watch run with color in Clion

Hello !
I'm on Clion 2021.2 and when I'm using cargo watch, I don't have any colors :(
So I see that I have to use --color=always  but how I pass it to the run command ?

Can't do that because I have this error : error: Found argument '--color' which wasn't expected, or isn't valid in this context

Ah, that can be a little confusing. The --color option must be on the cargo command, not the run subcommand. The -x option in cargo watch only prepends the word cargo, so you can use:
cargo watch -x '--color=always run'

or
cargo watch -- cargo --color=always run

if the latter is more obvious.

You're sure? I got the error: no such subcommand: `cargo

To which? I haven't actually tested this, no, as I'm on my phone, but I'll take a closer look when able

I'm also unfamiliar with Clion, so I don't know if that's playing into it; if you're able to test in a terminal as well, that would be helpful

Sorry, this is working good! I just had to remove the "cargo" because Clion already add it by default.
Thanks, buddy!


acBonus false positive

This is such an awesome module. On behalf of my entire table, thank you so much for the work you've put into it.
I think I'm getting a false positive for acBonus when potency runes are applied to armor. To Hit bonuses seem to get ignored when potency runes are applied to weapons, so I'm assuming this is indeed a false positive and not just intended behavior, but I definitely might be misunderstanding the intended workflow.
If there is anything I can do to help troubleshoot, please let me know. The reproduction steps seem pretty simple:

Add new armor from compendium to character sheet
Edit armor and add potency rune +1
Invest armor (doesn't trigger, notably, if the armor isn't invested)
Run Revitalize

Can you give 1.18.0 a try and see if that solves it for you?

Yes, it's working great! Thanks so much.


Support building docs against pip installed cupy

Also fix the issue that cupyx docs are not linked to GitHub.
Spin off from #4619 to fix build issues in https://github.com/chainer/chainer-test/pull/634

pfnCI, test this please.

Jenkins CI test (for commit 72bb00f04f2dd7ebb5d268b5610c8362fff0e34f, target branch master) succeeded!


[tmux] sometimes window fails to open

With the setting like this:
pane1 | pane2
------------------
      pane3

Running the following from pane2:
$ emacs &
[1] 29722
** (emacs:29722): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-6hbbbGBgHw: Connection refused

Happened with meld too.
$ meld . &
[1] 4657
$ /usr/lib/python2.7/dist-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: could not open display
  warnings.warn(str(e), _gtk.Warning)
/usr/bin/meld:155: GtkWarning: IA__gtk_icon_theme_get_for_screen: assertion 'GDK_IS_SCREEN (screen)' failed
  gtk.icon_theme_get_default().append_search_path(meld.paths.icon_dir())
Traceback (most recent call last):
  File "/usr/bin/meld", line 155, in <module>
    gtk.icon_theme_get_default().append_search_path(meld.paths.icon_dir())
AttributeError: 'NoneType' object has no attribute 'append_search_path'

I found that this can be worked around by the following for each terminal:
$ export DISPLAY=:0

Not sure but I ran these commands prior to the fix above:
sudo apt-get remove --purge xserver-xorg
sudo apt-get install xserver-xorg
sudo dpkg-reconfigure xserver-xorg

Not yet sure if I should add export DISPLAY=:0 to bash config. So for now I keep this ticket open.

On freshly installed machine,
$ meld . &
[1] 11128
n130s@tork-kudu1:~/link/github_repos/130s/compenv_ubuntu/config/bash$
(meld:11128): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Failed to connect to socket /tmp/dbus-zduub8N9uZ: Connection refused

(meld:11128): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Failed to connect to socket /tmp/dbus-zduub8N9uZ: Connection refused
Traceback (most recent call last):
  File "/usr/bin/meld", line 178, in <module>
    main()
  File "/usr/bin/meld", line 164, in main
    already_running, dbus_app = meld.dbus_service.setup(app)
  File "/usr/lib/meld/meld/dbus_service.py", line 56, in setup
    bus = dbus.SessionBus()
  File "/usr/lib/python2.7/dist-packages/dbus/_dbus.py", line 211, in __new__
    mainloop=mainloop)
  File "/usr/lib/python2.7/dist-packages/dbus/_dbus.py", line 100, in __new__
    bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop)
  File "/usr/lib/python2.7/dist-packages/dbus/bus.py", line 122, in __new__
    bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoServer: Failed to connect to socket /tmp/dbus-zduub8N9uZ: Connection refused

Should be fixed by #48
If not be sure to reopen.


Ignore null/falsy children in Masonry

This commit is the fix for https://github.com/cedricdelpoux/react-responsive-masonry/issues/42 issue.
In this, we simply ignore falsy children to get added in the columns array so that this doesn't break the UI.

@RavdeepSingh99 Thank you for this PR and sorry for the delay. I just want you clarify why the first condition.
Thank you


fix: remove settled transactions from duplicates

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/46372
PROPOSAL: https://github.com/Expensify/App/issues/46372#issuecomment-2255428812
Tests

Submit and pay an expense
Submit another two same expenses (same amount, date and merchant) as the paid expense
Go to transaction thread of any of the submitted expense
Click Review duplicates
Verify that: The paid expense does not have the button "Keep this one".
Click Keep this one on any of the unpaid expense
Verify that: The review process finished.

[x] Verify that no errors appear in the JS console

Offline tests

QA Steps

Submit and pay an expense
Submit another two same expenses (same amount, date and merchant) as the paid expense
Go to transaction thread of any of the submitted expense
Click Review duplicates
Verify that: The paid expense does not have the button "Keep this one".
Click Keep this one on any of the unpaid expense
Verify that: The review process finished.

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/user-attachments/assets/626fa59f-194e-42bc-a19b-e9bca061db98

Android: mWeb Chrome
https://github.com/user-attachments/assets/48d1d5b1-542c-4baa-af3e-ebd6b9aa891e

iOS: Native
https://github.com/user-attachments/assets/0c5de65c-6ad8-4f30-9c71-976d9184154a

iOS: mWeb Safari
https://github.com/user-attachments/assets/160c39aa-f2c9-48b0-9419-18673e5f9941

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/c3bdc0c6-ce53-49c1-a4b0-4562077471d8

MacOS: Desktop
https://github.com/user-attachments/assets/72a91007-45e4-4432-81ce-ef2637b3468a

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/techievivek in version: 9.0.27-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/roryabraham in version: 9.0.27-1 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Add  "Nat" tab to Firewall->Diagnostics->pfInfo menu

Hi!
Please, can you add "Nat" tab to pfInfo firewall diagnostics?
Thanks!

sure, although eventually we will probably replace the legacy page, it shouldn't hurt to add the raw nat output for now.

thank you!


about the training loss become nan

hi，nice work. But I meet some problems.
When I use the default config file and train the network on resllis3D, I found the loss became nan during training.
Has anyone else had a similar problem?
The first few records of the training are shown below：
2022-07-26 11:14:00,126 - mmseg - INFO - Iter [50/160000]       lr: 9.997e-03, eta: 11:13:45, time: 0.253, data_time: 0.010, memory: 17373, decode.loss_seg: 10.1304, decode.acc_seg: 21.6483, aux.loss_seg: 0.6783, aux.acc_seg: 21.7035, loss: 10.8088
2022-07-26 11:14:07,372 - mmseg - INFO - Iter [100/160000]      lr: 9.994e-03, eta: 8:49:53, time: 0.145, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 23.8205, aux.loss_seg: nan, aux.acc_seg: 31.7386, loss: nan
2022-07-26 11:14:14,590 - mmseg - INFO - Iter [150/160000]      lr: 9.992e-03, eta: 8:01:19, time: 0.144, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 24.1812, aux.loss_seg: nan, aux.acc_seg: 24.1812, loss: nan
2022-07-26 11:14:21,808 - mmseg - INFO - Iter [200/160000]      lr: 9.989e-03, eta: 7:37:00, time: 0.144, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 28.6286, aux.loss_seg: nan, aux.acc_seg: 28.6286, loss: nan
2022-07-26 11:14:29,039 - mmseg - INFO - Iter [250/160000]      lr: 9.986e-03, eta: 7:22:29, time: 0.145, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 24.6320, aux.loss_seg: nan, aux.acc_seg: 24.6320, loss: nan
2022-07-26 11:14:36,279 - mmseg - INFO - Iter [300/160000]      lr: 9.983e-03, eta: 7:12:52, time: 0.145, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 22.7228, aux.loss_seg: nan, aux.acc_seg: 22.7228, loss: nan
2022-07-26 11:14:43,525 - mmseg - INFO - Iter [350/160000]      lr: 9.981e-03, eta: 7:05:59, time: 0.145, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 24.7156, aux.loss_seg: nan, aux.acc_seg: 24.7156, loss: nan
2022-07-26 11:14:50,798 - mmseg - INFO - Iter [400/160000]      lr: 9.978e-03, eta: 7:00:59, time: 0.145, data_time: 0.001, memory: 17373, decode.loss_seg: nan, decode.acc_seg: 25.7715, aux.loss_seg: nan, aux.acc_seg: 25.7715, loss: nan

I found that it may be because the current code is single-card training, and the samples_per_gpu in rellis_group6.py needs to be changed to 2. Additionally, in the paper the authors say "In the ablations and comparisons, for fairness, we benchmark all our models at 240K iterations", but iterations are set as 160K in the ganav_group6_rellis.py. This is another a small question.

NAN again, after I changed the batch size from 1 to 2.
2022-07-26 11:33:46,558 - mmseg - INFO - Iter [50/240000]	lr: 9.998e-03, eta: 1 day, 0:43:21, time: 0.371, data_time: 0.005, memory: 17671, decode.loss_seg: 36.8659, decode.acc_seg: 19.6090, aux.loss_seg: 0.6523, aux.acc_seg: 28.1575, loss: 37.5183
2022-07-26 11:34:00,031 - mmseg - INFO - Iter [100/240000]	lr: 9.996e-03, eta: 21:20:13, time: 0.269, data_time: 0.002, memory: 17671, decode.loss_seg: 12.2197, decode.acc_seg: 31.8857, aux.loss_seg: 0.6118, aux.acc_seg: 34.3522, loss: 12.8315
2022-07-26 11:34:13,954 - mmseg - INFO - Iter [150/240000]	lr: 9.994e-03, eta: 20:24:18, time: 0.278, data_time: 0.002, memory: 17671, decode.loss_seg: 9.7978, decode.acc_seg: 29.3605, aux.loss_seg: 0.6117, aux.acc_seg: 34.8084, loss: 10.4095
2022-07-26 11:34:29,598 - mmseg - INFO - Iter [200/240000]	lr: 9.993e-03, eta: 20:30:37, time: 0.313, data_time: 0.002, memory: 17671, decode.loss_seg: 6.7851, decode.acc_seg: 28.4055, aux.loss_seg: 0.6025, aux.acc_seg: 32.5100, loss: 7.3876
2022-07-26 11:34:47,572 - mmseg - INFO - Iter [250/240000]	lr: 9.991e-03, eta: 21:11:34, time: 0.359, data_time: 0.002, memory: 17671, decode.loss_seg: 6.1998, decode.acc_seg: 28.8685, aux.loss_seg: 0.6131, aux.acc_seg: 37.6449, loss: 6.8129
2022-07-26 11:35:05,571 - mmseg - INFO - Iter [300/240000]	lr: 9.989e-03, eta: 21:39:06, time: 0.360, data_time: 0.002, memory: 17671, decode.loss_seg: 9.3100, decode.acc_seg: 30.8935, aux.loss_seg: 0.5739, aux.acc_seg: 40.4643, loss: 9.8840
2022-07-26 11:35:23,543 - mmseg - INFO - Iter [350/240000]	lr: 9.987e-03, eta: 21:58:23, time: 0.359, data_time: 0.002, memory: 17671, decode.loss_seg: 12.8057, decode.acc_seg: 26.7339, aux.loss_seg: 0.5988, aux.acc_seg: 36.8817, loss: 13.4045
2022-07-26 11:35:41,434 - mmseg - INFO - Iter [400/240000]	lr: 9.985e-03, eta: 22:11:56, time: 0.358, data_time: 0.002, memory: 17671, decode.loss_seg: 33.5382, decode.acc_seg: 39.3062, aux.loss_seg: 0.6051, aux.acc_seg: 43.8996, loss: 34.1433
2022-07-26 11:35:59,405 - mmseg - INFO - Iter [450/240000]	lr: 9.983e-03, eta: 22:23:08, time: 0.359, data_time: 0.002, memory: 17671, decode.loss_seg: 16.7517, decode.acc_seg: 24.2791, aux.loss_seg: 0.5761, aux.acc_seg: 41.6234, loss: 17.3278
2022-07-26 11:36:17,406 - mmseg - INFO - Iter [500/240000]	lr: 9.981e-03, eta: 22:32:17, time: 0.360, data_time: 0.002, memory: 17671, decode.loss_seg: 10.3142, decode.acc_seg: 30.3935, aux.loss_seg: 0.5899, aux.acc_seg: 39.1280, loss: 10.9040
2022-07-26 11:36:35,445 - mmseg - INFO - Iter [550/240000]	lr: 9.980e-03, eta: 22:39:59, time: 0.361, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 28.4942, aux.loss_seg: nan, aux.acc_seg: 39.3411, loss: nan
2022-07-26 11:36:51,019 - mmseg - INFO - Iter [600/240000]	lr: 9.978e-03, eta: 22:29:57, time: 0.311, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 20.4404, aux.loss_seg: nan, aux.acc_seg: 20.4404, loss: nan
2022-07-26 11:37:06,632 - mmseg - INFO - Iter [650/240000]	lr: 9.976e-03, eta: 22:21:40, time: 0.312, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 28.1801, aux.loss_seg: nan, aux.acc_seg: 28.1801, loss: nan
2022-07-26 11:37:22,231 - mmseg - INFO - Iter [700/240000]	lr: 9.974e-03, eta: 22:14:27, time: 0.312, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 26.7899, aux.loss_seg: nan, aux.acc_seg: 26.7899, loss: nan
2022-07-26 11:37:37,824 - mmseg - INFO - Iter [750/240000]	lr: 9.972e-03, eta: 22:08:08, time: 0.312, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 25.6708, aux.loss_seg: nan, aux.acc_seg: 25.6708, loss: nan
2022-07-26 11:37:53,442 - mmseg - INFO - Iter [800/240000]	lr: 9.970e-03, eta: 22:02:41, time: 0.312, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 25.0798, aux.loss_seg: nan, aux.acc_seg: 25.0798, loss: nan
2022-07-26 11:38:12,257 - mmseg - INFO - Iter [850/240000]	lr: 9.968e-03, eta: 22:12:51, time: 0.376, data_time: 0.002, memory: 17671, decode.loss_seg: nan, decode.acc_seg: 28.5087, aux.loss_seg: nan, aux.acc_seg: 28.5087, loss: nan

Have you tried to adjust/lower the learning rate accordingly? I have seen this issue but have it fixed after adjusting the learning rate.
You can adjust the number of iterations pretty easily and we only show one example of the config file we have, which has 160k iterations.

Have you tried to adjust/lower the learning rate accordingly? I have seen this issue but have it fixed after adjusting the learning rate.
You can adjust the number of iterations pretty easily and we only show one example of the config file we have, which has 160k iterations.

Thanks, I have tried to lower the learning rate from 0.01 to 0.005 and it works.
But I meet another problem. I set training iterations to 240k and eval moedl each 24k.(160k and 16k in default)
After the evaluation, the training peocess brke down.
The error messages are:
2022-07-26 13:37:16,619 - mmseg - INFO -
+------------+-------+-------+
| Class      | IoU   | Acc   |
+------------+-------+-------+
| background | 81.89 | 98.45 |
| L1         | 5.3   | 90.69 |
| L2         | 60.13 | 71.73 |
| L3         | 0.24  | 0.24  |
| non-Nav    | 36.88 | 45.8  |
| obstacle   | 57.56 | 72.78 |
+------------+-------+-------+
2022-07-26 13:37:16,620 - mmseg - INFO - Summary:
2022-07-26 13:37:16,620 - mmseg - INFO -
+--------+-------+-------+-------+
| Scope  | mIoU  | mAcc  | aAcc  |
+--------+-------+-------+-------+
| global | 40.33 | 63.28 | 70.93 |
+--------+-------+-------+-------+
2022-07-26 13:37:16,712 - mmseg - INFO - Exp name: ganav_group6_rellis.py
Traceback (most recent call last):
File "./tools/train.py", line 179, in
main()
File "./tools/train.py", line 175, in main
meta=meta)
File "/Data/xjy/yueye_project/GANav-offroad-main/mmseg/apis/train.py", line 118, in train_segmentor
runner.run(data_loaders, cfg.workflow)
File "/home/cyber/anaconda3/envs/ganav/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/home/cyber/anaconda3/envs/ganav/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 67, in train
self.call_hook('after_train_iter')
File "/home/cyber/anaconda3/envs/ganav/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
getattr(hook, fn_name)(self)
File "/home/cyber/anaconda3/envs/ganav/lib/python3.7/site-packages/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter
self.log(runner)
File "/home/cyber/anaconda3/envs/ganav/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 234, in log
self._log_info(log_dict, runner)
File "/home/cyber/anaconda3/envs/ganav/lib/python3.7/site-packages/mmcv/runner/hooks/logger/text.py", line 153, in _log_info
log_str += f'time: {log_dict["time"]:.3f}, '
KeyError: 'data_time'

I have fixed this problem. It may because the version of mmcv. I set the environment through:
conda install pytorch=1.10.0 torchvision cudatoolkit=11.3 -c pytorch
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html
You should modify the code in line 112 in mmseg/apis/train.py
from  runner.register_hook(eval_hook(val_dataloader, **eval_cfg))
to  runner.register_hook(eval_hook(val_dataloader, **eval_cfg), priority='LOW')

It’s possible that the MMCV (1.3.16) version do not need that flag, which I used. I’ll update that in the document and check backward compatibility for adding that flag. Thanks for the pointer!


common index between prover and verifier index

It would be nice to create a CommonIndex that is shared by both the prover and verifier indexes (domain, shifts, public input size). The rest of the prover index and the verifier index would then be mostly pre-computations IIUC.

we could use this issue to talk about refactoring the indexes in general. There is a number of improvements we've discussed in different places, but summarized:

ConstraintSystem should be split between:

a Circuit type that holds the gates, the public input size, the wiring, etc.
the rest should be moved to the prover index

the prover index should be split cleaned up between pre-computations, and other things
the verifier index should be split between static vs variable (as recommended by @rot256)
some parts of the prover and verifier index could be moved to a "common parameters"
some parts of the SRS not related to the PCS could be moved out of the SRS, and the SRS could be renamed something more user friendly (PcsParams?)


Redesign: Color scheme

Feature Description
Update general color scheme based on Figma.

[ ] Update all colors

Note that this includes the widely used background colors of the entire workspace.

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

This would then mean switching over to the new design system where the colors are already defined + removing the old theme usage fully + going through the colors to see if there are any that are hardcoded right now, yes?
There are quite a few places where something like rgba(theme.colors.bg.white, 0.1) etc is used, these should probably all be checked. It might take a bit more time to go through all these. WDYT?

Thinking a bit about it, the idea of adding it to the current theme is okay, but let's just call it system, so it's just:
const theme = {
  system: designSystemTheme,
  ...
};

Would that work for you?

Options as I see it:

Add system theme to current theme
Add current theme to system theme (e.g. as theme.old....)
Create a separate provider for system theme

I'm open to all three. 2 is probably too much work at this point though.

Some other work settled this.
We actually need to do item 2 above - or at least have the correct keys from the system theme. Because the design system components expect the theme to be the design system theme and will look for properties on it as if it is. This just made my work impossible in #6204, because our theme doesn't have a colors.border.focus property so the whole workspace component fails hard.
And this is going to clash for sure - because both the current theme and the new design system theme have a property located at theme.colors.fg.primary. Many other duplicates exist. This particular duplicate happens to be almost the same color (#E4E5E6 vs #EDEFEC), but that's not going to be the case for all colors nor all the other properties.
So, I think we need to move the old theme into a property called just that - and auto-replace all references to it. And then the theme will be:
const theme = {
  ...designSystemTheme
  old: {
    ...
  }
};

This will also make it clear, what needs to be deprecated in the future. We can even add a linter warning at some future date to warn about not using theme.old - and finally delete it when completely phased out.
And we need to prioritize this ticket higher, as it will be blocking (almost) all the other work.
I've asked @swissspidy to come up with a plan for how to have a temporary main branch for this work so we can work off of other work. If the two of you can solve that in the EU morning, that would be awesome! @swissspidy can also help you with the automated renaming required above. Most of it is probably a simple search-and-replace, but there might be some more tricky instances - especially in cases of useTheme().

Option 2 SGTM!

For the theme I'd recommend doing it the same way as was done for the dashboard so migrations can be done similarly.
See https://github.com/google/web-stories-wp/blob/fad69347fd5dc6f99f1831bca3f8e33542d30ad3/assets/src/dashboard/app/index.js#L124-L128 and the PR #5073

For the theme I'd recommend doing it the same way as was done for the dashboard so migrations can be done similarly.

Ok, will replace old in the PR to DEPRECATED_THEME for consistency.


MFCC extractor returns NaNs

I keep getting NaNs when I try to compute MFCC on my .wav audio file. I can email the audio file if needed as GitHub doesn't support uploading .wav files.
Code:
private static readonly MfccExtractor MfccExtractor = new MfccExtractor(
            16000,
            64,
            0.025,
            0.01,
            64,
            fftSize: 512,
            preEmphasis: 0.97,
            includeEnergy: true,
            dctType: "4",
            nonLinearity: NonLinearityType.LogE);
var samples = new WaveFile(new FileStream("./file.wav", FileMode.Open))[Channels.Left].Samples;
var mfccSamples  = MfccExtractor.ComputeFrom(samples);

If you're referring to the previously discussed nuance with -INF for log-energy, I haven't fixed it so far (because I don't think it's a bug at all). If you're talking about something else - then email me your file.

This is a separate issue. I'll email you the problematic file.

Are you sure you're processing exactly this file? I ran the extractor with your config and there are no NaNs here. At what time positions NaNs occur?

Just in-case anyone googles this too:
I was consistently getting a row of NANs at random frames samples when I was asynchronously running MFCC values on multiple samples.  Turns out I was building the extractor in the constructor and reusing the same instance across threads causing a race condition.
I've since fixed it by building a new extractor per task and it works fine.

Yes, feature extractors are not thread-safe by design. I'll add this important note to the corresponding page in wiki documenation.


Post results to current and to the new production enviroments

@josh right we can just switch over to the new squash-api-url then - that's fine with me.  Just to clarify that we don't need to update post-qa version, we can use the current version 1.3.1

@jhoblitt changes applied


February 2023 Meetup Registration

Happy February!
It's almost time for the February Rust Gamedev Meetup, it will be taking place this Saturday, the 11th of February! This meetup is an opportunity to show off cool things you've been working on in a less formal manner. This can take the form of a small demo, an overview of new features, or whatever you prefer :+1: If you haven't seen the meetup yet, you can check out last month's meet here: https://www.youtube.com/watch?v=iSu-9yKsCRY
If you have something cool, comment with the template below 🚀
Template
Project:

Discord username:

Estimated time:

Streaming Logistics
We use StreamYard to host the meetup. Using it is quite similar to another browser video chat tool, like Google Meet. We'll do a tech test before the meetup, so you can become familiar with how it is used.

(stealing some CCs from the newsletter, would love some new presenters!)
@kunos @anopara @cdsupina @ThousandthStar @johanhelsing @I3ck @uzkbwza @Indy2222 @TheGrimsey @Im-Oab @Dominux @johann2 @logicsoup @bcarruthers @I3ck @erlend-sh @pinkponk @Defernus @lee-orr

Graphite as usual! I'll edit this with a time estimate since it'll be either very short, or longer if we'll be announcing the product launch (if it's ready in time).

Thanks for considering me! I'm not yet ready to present at the Graphics meetup, but I've been thinking about it for a while and will let you know when I feel like I'm ready to present Idu. I was absent for most of last year, so I should really work on way more things to talk about something of substance.

Hey, thanks for the ping! My game isn't quite ready to be played though, as a lot of polishing needs to be done. I might present in a month or two.

Hey, thank you for the invitation as well. I would like to skip this month to have a bit more to present in March.

@logicsoup @ThousandthStar @Indy2222 we always love to see in-progress work, and so if you even just want to hop on to talk about your ideas or show off some behind-the-scenes, that would be cool too :)

Thanks for the invite, I'm interested in presenting, but I've been a bit unlucky with other things happening at the same time, and can't make it this time either. Maybe next month!

See everyone next month!