Could not find class dev.SASJavaExec

Hello,
I use the following sas code to run python program,
data _null_;
  length rtn_val 8;
  *** Python program takes working directory as first argument;
  python_pgm = "&WORK_DIR.\digitsdata_svm.py";
  python_arg1 = "&WORK_DIR";
  python_call = cat('"', trim(python_pgm), '" "', trim(python_arg1), '"');

  declare javaobj j("dev.SASJavaExec", "python", python_call);
  j.callIntMethod("executeProcess", rtn_val);
run;

but I got a error
Could not find class dev.SASJavaExec

I pretty sure I follow your instruction, but still the SAS could not find the dev.SASJavaExec class.

The most logically explanation is that you did not properly configure your CLASSPATH environment variable. Example to validate CLASSPATH here
You can set this in your SAS session by using the SET Option before you data step.
options set=classpath "C:\Path\to\SASJavaExec.class";

Hi @FriedEgg ,
Thank you for your quick reponse.
I have double checked my CLASSPATH, and also pass the VALIDATE JAVA CLASSPATH SAS Program.
But still I got this error.

and I attached the following code for your reference.

My SAS Program:

options set=classpath "C:\Users\c244032\Documents\SAS_Base_OpenSrcIntegration\bin";

%let WORK_DIR = C:\Users\c244032\Documents\SAS_Base_OpenSrcIntegration;
data _null_;
  length rtn_val 8;
  *** Python program takes working directory as first argument;
  python_pgm = "&WORK_DIR.\digitsdata_svm.py";
  python_arg1 = "&WORK_DIR";
  python_call = cat('"', trim(python_pgm), '" "', trim(python_arg1), '"');

  declare javaobj j("dev.SASJavaExec", "python", python_call);
  j.callIntMethod("executeProcess", rtn_val);
run;

My SAS_Base_OpenSrcIntegration path:

Judging solely by the pictures you have posted, the compiled class files appear to be in a different location in the first post you made than in the second.  There is still no reason to assume that the CLASSPATH not being set correctly is the item at fault here.  Can you verify that in you second post, the bin directory "C:\Users\c244032\Documents\SAS_Base_OpenSrcIntegration\bin" contains the subfolder "dev" which subsequently contain the file "SASJavaExec.class"
Try running the following in SAS and see what it says:
proc javainfo all; run;

@FriedEgg , Sorry for the late response as I was in vacation in last week. and also thanks very much for your quick response.
I double checked my directory, and it does contain dev with SASJavaExec.class.
Also I run the
Proc javainfo all; runn;

it shows
38   proc javainfo all; run;

PFS_TEMPLATE = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\qrpfstpt.xml
awt.toolkit = sun.awt.windows.WToolkit
file.encoding = GBK
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.headless = false
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = C:\PROGRA~1\SASHome\SASVER~1\eclipse\plugins\SASLAU~1.JAR
java.class.version = 51.0
java.endorsed.dirs = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\endorsed
java.ext.dirs = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\ext;C:\windows\Sun\Java\lib\ext
java.home = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre
java.io.tmpdir = C:\Users\c244032\AppData\Local\Temp\
java.library.path = C:\Program
Files\SASHome\SASFoundation\9.4\core\sasext;C:\windows\Sun\Java\bin;C:\windows\system32;C:\windows;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\W
indowsPowerShell\v1.0\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\C:\Users\c244032\Desktop\APP\SAS\Secure\ccme4;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program
Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\TDM-GCC-64\bin;C:\Program
Files\nodejs\;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;D:\APP\CTEX\UserData\miktex\bin;D:\APP\CTEX\MiKTeX\miktex\bin;D:\APP\CTEX\CTeX\ctex\bin;D:\APP\CTEX\CTeX\cct\bin;D:\APP\CTEX\CTeX\ty\bin;D:\APP\CTEX\Ghostscript\gs9.05\bin
;D:\APP\CTEX\GSview\gsview;D:\APP\CTEX\WinEdt;C:\Program Files\PuTTY\;C:\Users\c244032\Desktop\APP\SAS\x86\Secure\ccme4;D:\APP\Anaconda3;D:\APP\Anaconda3\Scripts;D:\APP\Anaconda3\Library\bin;C:\Program
Files\SASHome\SASFoundation\9.4\core\sasext;C:\Users\c244032\AppData\Local\Pandoc\;C:\Users\c244032\AppData\Local\rodeo\app-2.5.2\bin;C:\Program Files
(x86)\mingw-w64\mingw32\bin;C:\MinGW\bin;C:\MinGW\msys\1.0\bin;C:\Users\c244032\AppData\Roaming\npm;D:\APP\MTax\miktex\bin\x64\;C:\Program Files\Java\jdk1.8.0_131\bin;.
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.7.0_76-b13
java.security.auth.login.config = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.login.config
java.security.policy = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.policy
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 1.7
java.system.class.loader = com.sas.app.AppClassLoader
java.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
java.version = 1.7.0_76
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 1.7
java.vm.vendor = Oracle Corporation
java.vm.version = 24.76-b04
line.separator =

log4j.configuration = file:/C:/Program%20Files/SASHome/SASFoundation/9.4/tkjava/sasmisc/sas.log4j.properties
os.arch = amd64
os.name = Windows 7
os.version = 6.1
path.separator = ;
sas.app.class.path = C:\PROGRA~1\SASHome\SASVER~1\eclipse\plugins\tkjava.jar
sas.ext.config = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.java.ext.config
sas.home = C:\Program Files\SASHome
sas.jre.libjvm = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\bin\server\jvm.dll
sas.root = C:\Program Files\SASHome\SASFoundation\9.4
sas.tkjni =
sun.arch.data.model = 64
sun.awt.enableExtraMouseButtons = true
sun.boot.class.path =
C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\resources.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\rt.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\sunrsasign.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\jsse.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\jce.ja
r;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\charsets.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\jfr.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\classes
sun.boot.library.path = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = amd64
sun.desktop = windows
sun.io.unicode.encoding = UnicodeLittle
sun.jnu.encoding = GBK
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level = Service Pack 1
tkj.app.launch.config = C:\PROGRA~1\SASHome\SASVER~1\picklist
user.country = CN
user.dir = C:\Users\c244032
user.home = C:\Users\c244032
user.language = zh
user.name = C244032
user.script =
user.timezone = Asia/Shanghai
user.variant =

NOTE: ¡°PROCEDURE JAVAINFO¡±ËùÓÃÊ±¼ä£¨×Ü´¦ÀíÊ±¼ä£©:
      Êµ¼ÊÊ±¼ä          0.12 Ãë
      CPU Ê±¼ä          0.07 Ãë

@FriedEgg It is the same error that @77QingLiu faces which I am facing now. I have a doubt, in the document given for this: It has been specified that in the initial setup that Two Java classes: SASJavaExec.java and StreamManagerThread.java will get downloaded in the WORK_DIR which is not the case. Is that the problem ,that I am facing where the Error as "Could not find class dev.SASJavaExec"

@FriedEgg It is the same error that @77QingLiu faces which I am facing now. I have a doubt, in the document given for this: It has been specified that in the initial setup that Two Java classes: SASJavaExec.java and StreamManagerThread.java will get downloaded in the WORK_DIR which is not the case. Is that the problem ,that I am facing where the Error as "Could not find class dev.SASJavaExec"

@vishwath13 The pdf needs to be updated i.e., you no longer need two files as the content of StreamManagerThread.java was merged into SASJavaExec.java, hence you only need the latter.
There is a note in the readme that I am copying below that might help. Which version of Java did you use to compile?
NOTE: This project was compiled with Oracle JDK 1.7.0_25. We highly recommend using Java 7 to compile the java files in this project. When class files generated by some versions of Java 8 are used, the Base SAS Java Object fails with the following error:
ERROR: Could not find class dev.SASJavaExec at line 35 column 20. Please ensure that the CLASSPATH is correct. ERROR: DATA STEP Component Object failure. Aborted during the EXECUTION phase. java.lang.UnsupportedClassVersionError: dev/SASJavaExec : Unsupported major.minor version 52.0

@rmyneni Thank you so much. It is the same error except for the last line.

Where can I find the updated PDF Document and I tried using Oracle JDK 1.6.0_some version and later using 1.8.0_161. Should I specifically try with Java 7 version ? If so, Can you send me a link where I can download Java 7 version ? I was only able to get Java 8 versions available online.

@vishwath13 The PDF is accurate except for the list of classes you need -- that is, you only need SASJavaExec.java file. We have not tested with Java 6 so cannot comment on that but here is the Java 7 download site: http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html

Hi, Do you solution to this. kindly help

I am getting same issue. PLease see below
231
232  DATA null;
233      DECLARE JavaObj df("classfile/Classfile");
234      df.callDoubleMethod("comp", 1, 2, 3, r);
235      put r=;
236  RUN;
ERROR: Could not find class classfile/Classfile at line 233 column 23.  Please ensure that the
CLASSPATH is correct.
ERROR: DATA STEP Component Object failure.  Aborted during the EXECUTION phase.
java.lang.ClassNotFoundException: classfile.Classfile
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
NOTE: The SAS System stopped processing this step because of errors.
NOTE: DATA statement used (Total process time):
real time           0.01 seconds
cpu time            0.00 seconds
237
238  %reset_classpath;
NOTE: Setting Java classpath back to its original state: C:\Program Files\Java\jdk1.8.0_181\bin
239  PROC OPTIONS option=jreoptions;
240       RUN;
SAS (r) Proprietary Software Release 9.4  TS1M1

JREOPTIONS=(             -DPFS_TEMPLATE=C:\Program
Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\qrpfstpt.xml
-Djava.class.path=C:\PROGRA~1\SASHome\SASVER~1\eclipse\plugins\SASLAU~1.JAR
-Djava.security.auth.login.config=C:\Program
Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.login.config
-Djava.security.policy=C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.policy
-Djava.system.class.loader=com.sas.app.AppClassLoader
-Dlog4j.configuration=file:/C:/Program%20Files/SASHome/SASFoundation/9.4/tkjava/sasmisc/sas.log4j.prop
erties          -Dsas.app.class.path=C:\PROGRA~1\SASHome\SASVER~1\eclipse\plugins\tkjava.jar
-Dsas.ext.config=C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.java.ext.config
-Dsas.jre.libjvm=C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\bin\server\jvm.dll
-Dtkj.app.launch.config=C:\PROGRA~1\SASHome\SASVER~1\picklist          -Xms128m          -Xmx128m
)
Specifies the Java Runtime Environment options for SAS.
NOTE: PROCEDURE OPTIONS used (Total process time):
real time           0.09 seconds
cpu time            0.04 seconds
241
242  proc javainfo all; run;
PFS_TEMPLATE = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\qrpfstpt.xml
awt.toolkit = sun.awt.windows.WToolkit
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator =
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.headless = false
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = C:\PROGRA~1\SASHome\SASVER~1\eclipse\plugins\SASLAU~1.JAR
java.class.version = 51.0
java.endorsed.dirs = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\endorsed
java.ext.dirs = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre
java.io.tmpdir = C:\Users\PERSON~1\AppData\Local\Temp
java.library.path = C:\Program
Files\SASHome\SASFoundation\9.4\core\sasext;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:
Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;.
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.7.0_15-b03
java.security.auth.login.config = C:\Program
Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.login.config
java.security.policy = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.policy
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 1.7
java.system.class.loader = com.sas.app.AppClassLoader
java.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
java.version = 1.7.0_15
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 1.7
java.vm.vendor = Oracle Corporation
java.vm.version = 23.7-b01
line.separator =
log4j.configuration =
file:/C:/Program%20Files/SASHome/SASFoundation/9.4/tkjava/sasmisc/sas.log4j.properties
os.arch = amd64
os.name = Windows 7
os.version = 6.1
path.separator = ;
sas.app.class.path = C:\PROGRA~1\SASHome\SASVER~1\eclipse\plugins\tkjava.jar
sas.ext.config = C:\Program Files\SASHome\SASFoundation\9.4\tkjava\sasmisc\sas.java.ext.config
sas.home = C:\Program Files\SASHome
sas.jre.libjvm = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\bin\server\jvm.dll
sas.root = C:\Program Files\SASHome\SASFoundation\9.4
sas.tkjni =
sun.arch.data.model = 64
sun.awt.enableExtraMouseButtons = true
sun.boot.class.path =
C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\resources.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\rt.jar
;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\sunrsasign.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\jsse
.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\jce.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\charset
s.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\lib\jfr.jar;C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\classes
sun.boot.library.path = C:\PROGRA~1\SASHome\SASPRI~1\9.4\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = amd64
sun.desktop = windows
sun.io.unicode.encoding = UnicodeLittle
sun.jnu.encoding = Cp1252
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level = Service Pack 1
tkj.app.launch.config = C:\PROGRA~1\SASHome\SASVER~1\picklist
user.country = US
user.dir = C:\Users\Personal-PC
user.home = C:\Users\Personal-PC
user.language = en
user.name = Personal-PC
user.script =
user.timezone = Asia/Calcutta
user.variant =
NOTE: PROCEDURE JAVAINFO used (Total process time):
real time           0.10 seconds
cpu time            0.04 seconds

@praveensas are you failing running code from repo enlighten-integration/SAS_Base_OpenSrcIntegration or just trying to use JavaObj in Base SAS. Either way, you (1) need to compile Java class(es) that you want to use and (2) add them to the CLASSPATH of Base SAS installation. The pdf in this repo explains how to do that with an example -- see sections "Compiling the provided java classes" and "Setting the Java CLASSPATH"
Your code is using:
DECLARE JavaObj df("classfile/Classfile");
but it should be something like this:
DECLARE javaobj j("dev.SASJavaExec", "arg1");
where dev.SASJavaExec is the Java class with package extension that you want to execute and arg1 is the argument it takes. And you need to place this package structure (for example dev/SASJavaExec.class) in a directory and add that directory to the CLASSPATH of Base SAS installation.
Hope this helps!

hi, i faced similar issue.
i have compiled the file using java 7 and pasted SASJavaExec.class inside the recommended directory but it still fails.

this is the snipset of my code.

it throws me ERROR: Invalid Java Classpath.

hi,
i managed to run the check java class code perfectly without error but when i run the python code. it throws me this error.

Thank you for your assistance. managed to solve it after i change the python directory.

Thank you for your assistance. managed to solve it after i change the python directory.
however, there is another error as highlighted in yellow.

That error is coming directly from Python, maybe you don't have all the needed dependencies installed. Any line in the log that starts with "INFO: PROCESS OUTPUT >>>" is coming from Python.


Wrap Socket options

It looks like pub/sub doesn't work at the moment without first calling zmq_setsockopt on the subscriber. This is because the default for .subscribe sockets is to ignore all messages. We should start looking at wrapping zmq_setsockopt to get around this.
Providing a generic function as we have in the Context is not going to be so simple because, depending on the option name, different data types can be passed. I would suggest we either provide a method for each option name, or a method for each data type - the former seems neater but requires more boilerplate.

Nice suggestion. For example, ZMQ_AFFINITY  can be context.setAffinity. I was going to start working on it but I am currently on vacation without any access to a Mac. Please feel free to submit a pull request.

@jcockayne Any update on this one?

@jcockayne  Please also note the following new socket options in ZeroMQ 4.2

    ZMQ_HANDSHAKE_IVL
    ZMQ_SOCKS_PROXY
    ZMQ_XPUB_NODROP
    ZMQ_XPUB_MANUAL
    ZMQ_XPUB_WELCOME_MSG
    ZMQ_STREAM_NOTIFY
    ZMQ_INVERT_MATCHING
    ZMQ_HEARTBEAT_IVL
    ZMQ_HEARTBEAT_TTL
    ZMQ_HEARTBEAT_TIMEOUT
    ZMQ_XPUB_VERBOSER
    ZMQ_CONNECT_TIMEOUT
    ZMQ_TCP_MAXRT
    ZMQ_THREAD_SAFE
    ZMQ_MULTICAST_MAXTPDU
    ZMQ_VMCI_BUFFER_SIZE
    ZMQ_VMCI_BUFFER_MIN_SIZE
    ZMQ_VMCI_BUFFER_MAX_SIZE
    ZMQ_VMCI_CONNECT_TIMEOUT
    ZMQ_USE_FD

Yes, sorry I am getting there! I did a large number of them this weekend (see my fork, these are predominantly those described in the old ZMQ docs), but as you pointed out there are more that I have not yet wrapped. I'll try to find some time to do this over the next few days.

@jcockayne I saw your fork earlier. Great work :+1:

@jcockayne  What about I manually merge https://github.com/jcockayne/SwiftyZeroMQ/commit/4fa93d81d2c79ae6fc035590c138d236749cd532? 👍

Sure - or I can just submit a pull request if that's easier? Even though it's not yet completed a lot of the important ones are wrapped and the pattern is there!

Go for it 👍

I'm very interested in the progress of this PR, thanks for giving this issue some attention! Any chance there will be a pub-sub example added to this repo once the PR is merged?

@jcockayne I will be integrating the changes in your PR with master.
@erparker thanks for the interest, a pub-sub example will be added soon 👍

@erparker a pub-sub example is released with 1.0.24
@jcockayne Thanks for your contribution.
Have fun.


Feature: Knob-less QoS with fq_codel/cake and something like OpenWRT's "SQM"

Given the amazing success of OpenWRT with fq_codel and SQM at beating back bufferbloat and providing low latency across flows with having to tune: http://www.bufferbloat.net/projects/codel/wiki
I'd like to move some of my company's pfsense boxes over to a distro that uses something like this. Right now IPFire (being linux-based) is able to do this pretty easily, but I would love to use OPNsense.
This seems seriously non-trivial to do in FreeBSD given the chatter in the pfsense community about this.

We're using a different system for traffic shaping and QoS (ipfw dummynet), which doesn't contain the codel algorithm.
There is a custom patch available for ALTQ/pf (which is in pfSense), but won't match our codebase.
In ipfw/dummynet there also are some options for scheduling, which are not in our UI at the moment, but which should be a more logical approach in our case.
https://www.freebsd.org/cgi/man.cgi?ipfw(8)#TRAFFIC	SHAPER_(DUMMYNET)_CONFIGURATION

Right -  Would you give any consideration to fq_codel (not really the same as codel, see the bufferbloat.net link above), or cake? From my experience in all sorts of shaping / QoS systems, they are significantly ahead of the rest of the field in out-of-the-box effectiveness:
https://indico.uknof.org.uk/getFile.py/access?contribId=3&resId=0&materialId=slides&confId=27

I really don't see ALTQ/CODEL (it will be in FreeBSD 11 courtesy of pfSense) combination lift off as ALTQ is (and most likely will remain) disabled in FreeBSD GENERIC. OpenBSD removed ALTQ as well some time ago.
I've read the page you provided and a bit more on the topic (thanks btw). There seems to be dual-licensed code that could make its way into FreeBSD in another way. The work looks very promising. I also like the zero-config approach, although in theory this shouldn't be a subsystem, it should be a holistic switch that covers all traffic flowing through the box (or an interface). As such it may have side effects with an enabled traffic shaper, but it's better than having to deal with "either this or that, not both" scenarios. Or at least that's how the GUI should handle it, right? :)

Honestly, I'm not sure - it's dual licensed but from what I've read (I'm no expert on kernels or low-level nets code) the port to FreeBSD is not easy.
With regard to the GUI side, having something like OpenWRT's SQM would be my personal idea for my team's deployments.

Looks like there is a Comcast-sponsored student working on getting fq_codel into FreeBSD dummynet:
http://lists.freebsd.org/pipermail/freebsd-net/2015-September/043443.html

Great news indeed, this probably won't make it into FreeBSD 11.0 in time, but I'm sure we can backport or wait for 11.1. The OPNsense traffic shaper code is basically ready for this as is now.

Very cool. It's hard to overstate just how impressive fq_codel w/ BFQ is in action - I highly recommend spinning up OpenWRT and seeing it in action with SQM limited to 95% of bandwidth.

hot off the press ;)
https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html

Wow! I'm impressed they cranked this out that quickly...

We also have a test kernel based on the latest OPNsense code ;)
In case anyone wants to try it:
# opnsense-update -bkr 16.1.3-aqm && /usr/local/etc/rc.reboot

Here's the doc to operate AQM on the command line...
http://caia.swin.edu.au/freebsd/aqm/patches/README-0.1.txt

Great job! Seems to work OK, very interested in how it gets integrated into GUI - take a look at OpenWRT sqm in luci, it's super simple and works great.

Applause! Benchmarks wanted. (try using https://github.com/tohojo/flent )
Is the -0.2 patch incorporated yet? That fixed a few problems, notably one with ecn handling.

@dtaht not yet shipped, but it's in the repo already https://github.com/opnsense/src/commit/fb03383d56576669d771259e40ad83b53a992e3b
I will push another test build soon enough. Preliminary compile looked good yesterday
Thanks for the link, will benchmark what I can from here :)

Test base/kernel with v0.2 is up for amd64:
# opnsense-update -bkr 16.1.9-aqm && /usr/local/etc/rc.reboot

performed some simple tests to see how it works (using v0.1), thanks @dtaht for the tip of using flent.
All tests performed using the following command:
flent rrul -p all_scaled -l 60 -H hostname -t "Title" -o filename.png

Test 1: ipfw enabled, but not passed through dummynet

Test 2: dummynet enabled, using default Weighted Fair Queueing (wf2q+)

Test 3: CoDel enabled, using defaults

Test 4: FQ-CoDel enabled, using defaults

Initial work is delivered with 16.1.12 today. I'm going to close this ticket now.
The work will continue on our end, e.g. AQM v0.2 will be merged shortly after a bit more testing.
Feel free to discuss this ticket / its results further and new tickets for individual improvements and bugs so we can track them independently.
Thank you all for your input, testing and help. :)

your "codel" result (test 3) doesn't make any sense. You should have seen 10-20ms latency on this test with pure codel. Also, you can turn off log scales when generating test results in flent....

also, @AdSchellevis were you explicitly shaping to 400mbit's or trying to run at line rate? Certainly shaping eats a great deal of cpu, and we don't know how fast BSD boxes can actually forward packets at line rate at all at this point, no matter the aqm/fq technology in play.
(hint, you can produce a comparison test in flent-gui by loading up all the *.flent.gz files and selecting "Data->add other open files) - bar charts, etc....)
PS: If you could stick up your flent.gz files somewhere I could get them, I could do a more full writeup elsewhere (blog.cerowrt.org probalby)
Thx VERY much for showing classic FQ result, also.

@dtaht I'm not sure why you expect 10-20ms, without codel it was around 10-20ms under stress and dropped to 2ms with codel enabled, maybe I'm missing something or misinterpret the reading.
While testing, I had 2 different pipes (1 up, 1 down) limited to the max line speed (1Gbps), so shaping could certainly have impacted my performance a bit.
I deleted the *.flent.gz files from my machine, the measurements are probably not the best in the world. I can rerun the tests later under approx the same circumstances and send them to you then.
Can you provide me with an email address to send the files (or a link) to?
Thanks for the hint, I will certainly try the flent-gui too.

From eyeballing the tests you only got 400mbit in both directions on the gbit link, when could crack about 880 in both directions. Your first test without anything in play (again, from eyeballing, there's a bar chart and totals chart in the flent-gui that makes it easier to read) hit 680/600 or so. So what you were measuring was loss elsewhere in the stack. Probably.
Try shaping to 100mbit on both sides of the link to see a difference between codel and fq_codel.

Yesterday I wasn't able to redo some testing, a bit too busy. I will try to run the 100Mbps test like you suggested next week (yes, my measurements where well below max, probably old / slow switches in between, a bit too busy to build a decent test setup ;) ).
If you drop me an email (ad at project domain), I will send you the results next week.

one result I think you are showing is that fq_codel tends to drop stuff sanely when router cpu is overloaded. ;) I look forward to your results and I dropped you an email a few minutes ago.

I would like to announce that we released Dummynet AQM version 0.2.1 (CoDel, FQ-CoDel, PIE and FQ-PIE) which includes important bugs fixing. I highly recommend to upgrade to this version.

@RasoolAlSaadi thank you, added via https://github.com/opnsense/src/commit/74aa1a19f899dfff548c18ad25f4844b0105755c

So with the recent 16.1.14 does one just create a pipe with FlowQueue-CoDel as the scheduler and with Enable CoDel checked to use FQ-CoDel? Or does the Enable CoDel checkbox interfere with it?
Thanks for the great work guys!

@skarekrow you can either use FlowQueue-CoDel or use "Enable Codel" on another scheduling mechanism (for example the default wf2q+). I don't think the checkbox actually does anything on  FlowQueue-CoDel

@AdSchellevis Ah, thank you sir :)

Is cake available yet? Seems to be a marked improvement over fq_codel @fichtner

Nope, I asked the dev's of CoDel implementation 2 months ago, no plans to implement it.

Heri Sim<EMAIL_ADDRESS>writes:

Is cake available yet? Seems to be a marked improvement over fq_codel
@fichtner

cake is really dependent on a nsec clock to work. BSD was only msec
when I last understood it.?
there is a ns3 model in progress if that helps any.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


undefined module of 'versioning' during migration from 0.7.6 to 0.11.3

Issue Summary
I downloaded ghost v0.11.3 and was trying to copy the files and overwrite my existing installation of 0.7.6.
After npm install, when I run npm start --production, it errors out saying
"canMigrateFromVersion" is not found in undefined.
After looking at the code, I found that if I change the line
versioning = require('../schema').versioning,

in core/server/data/migration/update.js to
versioning = require('..schema/versioning')

then the code can continue running. I had to do the same kind of syntax modification a few other places to finally make the database migration code work, and run ghost 0.11.3.
I'm running Ghost on Debian Testing distribution with Linux kernel version <IP_ADDRESS> with node v4.2.1, not sure whether the Testing distribution is problematic here, but from the code change to make the migration work, I don't think it's related to kernel version as long as the node version I'm running is supported.
Steps to Reproduce
Just install 0.7.6, put in some post and overwrite with 0.11.3.
Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?
I expect the upgrade would succeed without me changing the js code.
Technical details:

Ghost Version: 0.7.6 -> 0.11.3
Node Version: 4.2.1
Browser/OS: Debian Stetch Testing
Database: Sqlite 3

Hey @1919yuan Sorry to hear you are having trouble.
You have updated from 0.7.6 and this version contains a schema.js file.
See https://github.com/TryGhost/Ghost/tree/0.7.6/core/server/data
It looks to me this file is still left in your file tree. And this file does not contain an object called versioning. That's why your fix worked. Glad you figured out a solution to update to 0.11.3 successfully.
I assume something went wrong when you deleted the old files (see http://support.ghost.org/how-to-upgrade/#3.-delete-the-old-files)
If you have further questions can you please swing by our slack channel and ask for help there? Thanks 👍


Other languages

My search works in English but not Japanese. The index.json file has Japanese entries, but the search doesn't return anything when searching for Japanese strings. Can anyone reproduce this and/or suggest a fix?

The underlying search library, lunr.js, only supports English. So unfortunately you will not be able to search with Japanese text.

Ah, ok, thanks for the info.


Bug: npm package is missing dist folder

The released versions 1.0.7 and 1.0.8 on NPM do not have a dist folder!

👀 Take  a look

Hi,
Thank you for reporting the issue.
I have uploaded a new version 1.0.9 that contains the dist folder.
Please check it out:
<EMAIL_ADDRESS>If something still not ok, please reopen the ticket.


Error when using deoplete

NVIM v0.2.1-279-g0183830
deoplete.nvim at commit c8fd11834a76c58ec976be613b3e1c99a97091f8
vim-racer at commit 92c3e2b57e60c3d4f0102d1d587ddc762e662f60
racer 2.0.8
Python 3.5.2
neovim python module 0.1.13
rustc 1.17.0 (56124baa9 2017-04-24)
cargo 0.18.0 (fe7b0cdcf 2017-04-24)

In certain situations I get an error when  adding a new global constant.
The error is

[deoplete] Traceback (most recent call last):
[deoplete]   File "/home/adno/.config/nvim/bundle/deoplete.nvim/rplugin/python3/deoplete/deoplete.py", line 119, in gather_results
[deoplete]     ctx['candidates'] = source.gather_candidates(ctx)
[deoplete]   File "/home/adno/.config/nvim/bundle/vim-racer/rplugin/python3/deoplete/sources/racer.py", line 71, in gather_candidates
[deoplete]     kind = typeMap.get(completions[4], '')
[deoplete] IndexError: list index out of range
[deoplete] Could not get completions from: racer.  Use :messages for error details.

Steps to reproduce using nvim -u minimal.vim
minimal.vim
call plug#begin()
    Plug 'Shougo/deoplete.nvim'
    Plug 'racer-rust/vim-racer'
call plug#end()

let g:deoplete#enable_at_startup = 1

Steps
Create a new project
cargo new tmp
cd tmp

Edit the generated lib.rs
nvim -u PATH/TO/minimal.vim src/lib.rs

Add a constant to the file. For example, the file might then look like this:
#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
    }
}

pub const A_CONST: u8 = 2;

In the line above the const, try to add another constant. For example:
#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
    }
}

pub const SECOND_CONST: u8 = 1;
pub const A_CONST: u8 = 2;

You will get an error after writing just pub const SE (since that is the default minimum length for completion).

I added a debug print line to the gather_candidates function of racer.py. Apparently when the error occurs it is because completions is a single item array containing just the identifier of the partially typed second constant.

Actually, this looks like its a racer problem. Calling racer manually to complete the position (racer complete 8 11 src/lib.rs) results in

PREFIX 76,77,S
MATCH SE
pub const A_CONST,8,10,src/lib.rs,Const,pub const SE
MATCH Send,43,17,/home/adno/.multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/../libcore/marker.rs,Trait,pub unsafe trait Send
MATCH Sized,92,10,/home/adno/.multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/../libcore/marker.rs,Trait,pub trait Sized
MATCH Sync,346,17,/home/adno/.multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/../libcore/marker.rs,Trait,pub unsafe trait Sync
MATCH Some,165,4,/home/adno/.multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/../libcore/option.rs,EnumVariant,Some(#[stable(feature = "rust1", since = "1.0.0")] T),
MATCH SliceConcatExt,1247,10,/home/adno/.multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/../libcollections/slice.rs,Trait,pub trait SliceConcatExt<T: ?Sized>
MATCH String,263,11,/home/adno/.multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/../libcollections/string.rs,Struct,pub struct String
END

Note the line break between the "MATCH SE" and the "pub const A_CONST,8,10,src/lib.rs,Const,pub const SE". vim-racer probably isn't capturing it all as a single entry because the parts are on separate lines.

Adding println!s to racer confirmed that the issue is indeed caused by racer.
I opened an issue there (https://github.com/racer-rust/racer/issues/746).

OK.  I wait until the problem is fixed.

Any update on this? Expiriencing the same issue

No response from upstream yet.
So same problem exists in racer.

From taking a look at the issue opened on racer it looks like this issue has been resolved upstream 14 days ago

OK.


updating the readme with openscap details

I forgot to do this before..

@pweil- can you review? (Also the text if needs to be clearer, etc.)

wording looks good to me

@enoodle did you revert this back to use port 5000? Why?

I am sorry, I fixed a comment from a different computer and forgot to pull before. I will fix that again.


pluginlib doesn't compile on debian:stretch

ROS distro: lunar
Debian: stretch
Default boost version through apt: 1.62
Building ROS Lunar from source as directed here: http://wiki.ros.org/Installation/Source. At the step ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release, during compilation of pluginlib, the executable plugin_tool fails at the linking step with the following error:

librosconsole.so: undefined reference to boost::re_detail::cpp_regex_traits_implementation<char>::transform[abi:cxx11](char const*, char const*) const' librosconsole.so: undefined reference to boost::re_detail::raise_runtime_error(std::runtime_error const&)'
librosconsole.so: undefined reference to boost::re_detail::put_mem_block(void*)' librosconsole.so: undefined reference to boost::re_detail::perl_matcher<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > >, boost::regex_traits<char, boost::cpp_regex_traits > >::construct_init(boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits > > const&, boost::regex_constants::_match_flags)'
librosconsole.so: undefined reference to boost::re_detail::get_mem_block()' librosconsole.so: undefined reference to boost::re_detail::verify_options(unsigned int, boost::regex_constants::_match_flags)'
librosconsole.so: undefined reference to boost::re_detail::get_default_error_string(boost::regex_constants::error_type)' librosconsole.so: undefined reference to boost::re_detail::cpp_regex_traits_implementation::transform_primary[abi:cxx11](char const*, char const*) const'
collect2: error: ld returned 1 exit status

Initially I thought it was a bug in pluginlib/rosconsole. But on looking at other similar cases with other libraries, it seems like it could be a version configuration bug. Is it possible the default version of boost is higher than what is required?

Confirmed it works with boost 1.54, so between that and 1.62 something was added that breaks the build. Maybe the rosdep file should be modified to include a specific version number for debian.

@2ros0 what does your workspace contain? I was just able to build pluginlib in a stretch container following the instructions with three modifications:

Installed rosdep from pip to circumvent an error
Applied --rosdistro lunar --os=debian:stretch to my rosdep invocation since my container wasn't being auto-detected.
Built with  ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release --only-pkg-with-deps pluginlib to build only up to pluginlib.

Was a case of multiple installed boost versions. The remote device had an older version of boost preinstalled with some cmake files causing a configuration error. apt's libboost-all-dev (currently 1.62) installed through rosdep works fine. Thanks for checking.


Ability to mirror Github repository

Hello,
would be nice if gogs could have ability to mirror/backup existing Github repository.
Including issues and pull requests (and maybe wiki pages).
With option to set how often it should be updated.
Best regards

Sorry to necrobump but this is awesome +1 great job :D


Replace fread with mmap

Replace fread with mmap method
Performance comparison
Load Linux image:

fread method:<PHONE_NUMBER>3709535686 clock
mmap method:                                       157 clock

Check https://github.com/jserv/semu/blob/master/CONTRIBUTING.md carefully and ensure the consistent style.

Any update?

yes, coding style was changed, but  i don't know why i need to use atexit(3) to perform the munmap operation, is this for error handling? like the following code:
void cleanup(*addr, size_t len)
{
    munmap(...)
}
static void read_file_into_ram(char **ram_loc, const char *name)
{
...
    if (*ram_loc == MAP_FAILED) {
        fprintf(stderr, "mmap failed\n");
        atexit(cleanup);
    }
}

yes, coding style was changed, but i don't know why i need to use atexit(3) to perform the munmap operation, is this for error handling? like the following code:

No, atexit is invoked right after mmap succeeds. It is used for normal execution path rather than the abnormal one. i.e., exception handling.

Superseded by commit 944a51d612aef623b51ebcdd4956336bae61e0a6

Thank @zoanana990 for contributing.


Reimplement ensemble majority vote

The current implementation is pretty slow. I suggest experimenting with this SO answer (use numpy instead of Counter).
The only limitation is that only non-negative integers are supported, however, we can always check this condition and fallback to Counter.
Related file: https://github.com/Cognexa/cxflow/blob/dev/cxflow/models/ensemble_model.py

np should be indeed faster
➜ red@red ~
python -m timeit 'import numpy as np;from collections import Counter;Counter(np.random.randint(0,100,1000)).most_common();'
10000 loops, best of 3: 128 usec per loop
➜ red@red ~
python -m timeit 'import numpy as np;from collections import Counter;np.argmax(np.bincount(np.random.randint(0,100,1000)));'
100000 loops, best of 3: 11.6 usec per loop

On the other hand:

np.bincount is limited to 1-dim integer arrays
we can expect millions (of examples) x units (of models) x tens (of classes) hence one call of the major_vote function would be on a quite small object, i.e. the following measurement fits better

@petrbel I would suggest closing this issue. I do not think we can speed it up significantly. The major vote aggregation is simply not suitable for a large amount of examples (pixels in your case). Btw, 10 Isletnet images have roughly 30M millions of pixels (examples).

ok, closing


[Proposal] The ability to add "view composer" functionality or variables available to every page.

I find that if I have a long tree of nested objects like this:
subject > topic > course > lesson

I have to include the subject, topic, and course in every lesson file. What if this was stored in the config file somehow like this?
'collections' => [
   'lessons' => [
       'path' => '...',
       'sort'  => '...',
       'variables' => [
             'subject' => '...',
             'topic' => '...',
             'course' => '...',
        ]
   ],
   ...
]

That way I don't have repeat all these variables in every single lesson file. Thoughts?

How would you determine the value for each of subject, topic, and course in config.php? Seems that those would vary by file.
What you're doing is emulating nested collections, something that isn't fully supported at the moment. Ultimately, it would be best to allow collection items to live in subdirectories, and set them up in config like:
'collections' => [
    'subjects' => [
        'path' => '...',
        'collections' => [
            'topics' => [
                'path' => '...',
                'collections' => [
                    'courses' ...
                ],
            ],
        ],
    ],
]

Then, a collection item's position within a subdirectory (or subdirectories) would take the place of the extra subject/topic/course variables that you're currently specifying in the YAML header.

Yes, exactly! that is what I'm thinking. I find that I am resulting to that directory structure anyway to stay organized with all of my lessons. So instead of repeating myself with the variables it would be nice to setup the directory structure in config file, so I can remove all those variables from the files and update them in a single place.

Assuming subjects, topics, courses, and lessons` are all collections, you could avoid duplicating all the extra variables in your markdown files by just specifying the immediate parent; it's always possible to reference up the hierarchy by accessing that parent collection.
So, for example:

lessons would only need a course variable, which would be the filename of the parent course (e.g. css-basics).
courses would only need a topic variable, with the filename of its parent (e.g. css)
topics would only need a subject variable, with the filename of its parent

Collection items are keyed by their filenames (not including any extension like .md or .blade.php. So, from within lessons, you can access the parent coursecollection item via$courses->get($page->course)`.
Let's say you wanted to output breadcrumbs for a lesson; you can traverse up the hierarchy in your Blade template, and get each parent collection item, like this:
@php
    $course = $courses->get($page->course);
    $topic = $topics->get($course->topic);
    $subject = $subjects->get($topic->subject);
@endphp

getUrl() }}">{{ $subject->title }} /
getUrl() }}">{{ $topic->title }} /
getUrl() }}">{{ $course->title }}

You could even generalize these variables, and just name them all parent.

I ran into a similar issue...
Imagine you have the following:
Cars > Make > Model
You can do a "cars" page, you can do a collection of makes, but you can't do a nested collection of models.
So, for example, if I put a file in _cars/_fiat/panda.blade.php then I would expect to be able to reach it by visiting /cars/fiat/panda.
However, this currently appears to ignore the middle layer and will be called /cars/panda.


#include <bits/stdc++.h>

Need to figure out how to get #include <bits/stdc++.h> import working in vscode so I don't have to import a million packages every time.

fixed it but there is still an error message in vscode. still compiles tho :(

fixed it but there is still an error message in vscode. still compiles tho :(


OAuthInput throws when turn.activity.text is null

Version
4.10.5
Describe the bug
In msteams, when an OAuthInput dialog is active and waiting for user input, if the user submits a response that is not an auth token and has a null value for turn.activity.text, then OAuthInput will throw with the following exception:
System.ArgumentNullException: Value cannot be null. (Parameter 'input')
   at System.Text.RegularExpressions.Regex.Match(String input)
   at Microsoft.Bot.Builder.Dialogs.Adaptive.Input.OAuthInput.RecognizeTokenAsync(DialogContext dc, CancellationToken cancellationToken)

I believe this is due to a missing null check when checking the message input for the presence of a magic code.
To Reproduce
Steps to reproduce the behavior:
Write an AdaptiveDialog containing any OAuthInput, for example:
new OAuthInput
{
    ConnectionName = this.connectionName,
    Text = "Before I can perform this operation, I need permission to validate your identity.",
    Title = "Sign-in required",
    Property = "$authToken",
    InvalidPrompt = new ActivityTemplate("Sorry, please try again."),
    AllowInterruptions = "turn.recognized.intent == 'Cancel' || turn.recognized.intent == 'Help'"
},

Run the bot in msteams, and trigger the auth input dialog. When the "Sign-in required" button shows up in Teams, instead of clicking on it, submit a message that would result in a null value for turn.activity.text. There are several ways to do this:

Click a button on an adaptive card (not the sign-in button) that sends an empty text value, for example:

{
    "type": "Action.Submit",
    "title": "Title",
    "data": {
        "msteams": {
            "type": "messageBack",
            "displayText": "nothing much",
            "text": "",
            "value": {"InputSource": "MyCard"}
        }
    }
}

Send a Teams emoji with no additional text
Use the rich text editor in Teams, select the 'quoted message' option, then send an empty quote box

Expected behavior
The OAuthInput should display its InvalidPrompt message and re-prompt the user (if MaxTurnCount has not been reached).
As an aside, the Validations property inherited from InputDialog does not seem to be implemented in OAuthInput, so it is not possible to add a validation checking if turn.activity.text is null.

@lucaism looking into this. Currently I have a bot plugged into Teams that shows me the OathInput prompt, however currently it doesn't even do anything  even if I do something standard as clicking it...

^ stays frozen there
Could you share more code with me, please, such as what action(s) you take after you prompt using the OAuthInput?

Actually, nvm, managed to get it unfrozen. For some reason when I was running the bot in VS Code rather than VS it would run the bot, but wouldn't do anything once I clicked the Sign-in button (problem to investigate later...)
Managed to reproduce error @lucaism reported.

Microsoft.Bot.Builder.Integration.AspNet.Core.BotFrameworkHttpAdapter: Error: Exception caught : Value cannot be null. (Parameter 'input')

I'll discuss with the team what the next steps here should be. Thank you for raising the issue

Also yes, it does break in the place you described in your original post in OAuthPrompt.RecognizeTokenAsync:


Nested pydantic request model raising ValueError returns 500 instead of 4xx

First of all, thanks for the great work with FastAPI 🥇  It's very much appreciated!
I think the problem I have is more related to FastAPI than pydantic because I manage to get the intended behavior when using the same pydantic models outside of the FastAPI context.
Example
I have something like:
from typing import List
import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel, validator

app = FastAPI()

class Child(BaseModel):
    name: str

    @validator("name")
    def _validate_name(cls, value):
        raise ValueError("wrong name!")

class Parent(BaseModel):
    child: Child
    foo: str

    @validator("foo")
    def _validate_foo(cls, value, values):
        name = values["child"].name
        if value != name:
            raise ValueError("name should match")
        return value

class SubModel(BaseModel):
    parents: List[Parent]

class RequestPayload(BaseModel):
    payload: List[SubModel]

@app.post("/")
def root(payload: RequestPayload):
    return {"foo": "bar"}

if __name__ == "__main__":
    uvicorn.run(app, host="<IP_ADDRESS>", port=9999)

Description
When calling the above endpoint as follows: curl -X POST -H "Content-Type: application/json" http://<IP_ADDRESS>:9999/ -d '{"payload": [{"parents": [{"child": {"name": "child"}, "foo": "bar"}]}]}' I get Internal Server Error as a response, failing in the validate_foo with KeyError: 'child'. I expected the API to return 4xx and and the value error raised in the validation of the Child model. Shouldn't the error have stopped the validation altogether having the API return 4xx?
Am I doing something wrong?
Thanks for taking the time to read my question ❤
Environment

OS: MacOS
FastAPI Version: 0.65.2
Pydantic Version: 1.8.2

Problem
The path of your data is this:

Data gets on the root endpoint.
RequestPayload :heavy_check_mark:
List[SubModel] :heavy_check_mark:
Parent :standing_person:
Validate child :open_mouth:
Find out that the field name of Child is invalid! :cry:
Try to use values["child"], but values doesn't have child because it was not valid. :wine_glass:

How to solve this issue
Always make sure that you have the key you want from values:
from typing import List
import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel, validator

app = FastAPI()

class Child(BaseModel):
    name: str

#    @validator("name")
#    def _validate_name(cls, value):
#        raise ValueError("wrong name!")

class Parent(BaseModel):
    child: Child
    foo: str

    @validator("foo")
    def _validate_foo(cls, value, values):
        # print(values)  # This print here can be valuable for debugging! :)
        if "child" in values and values["child"].name != value:
            raise ValueError("name should match")
        return value

class SubModel(BaseModel):
    parents: List[Parent]

class RequestPayload(BaseModel):
    payload: List[SubModel]

@app.post("/")
def root(payload: RequestPayload):
    return {"foo": "bar"}

if __name__ == "__main__":
    uvicorn.run(app, host="<IP_ADDRESS>", port=9999)

You can check that all examples on Pydantic documentation follow what I just recommended.

Thanks for taking the time to read my question :heart:

You're welcome! :smile:

Thanks for the answer 🙂  I will make the changes accordingly, but shouldn't the ValueError get propagated in Child? Just trying to understand the behavior

Well, if you don't get the KeyError, it will be. 🤔

Ah, that's correct 🙂  thanks for the quick reply, this is solved now ❤


EACCES: permission denied during npm install

Hi,
I'm having an issue installing<EMAIL_ADDRESS>using npm 6.14.6.
Here is the trace of the error :
<EMAIL_ADDRESS>preinstall /usr/local/lib/node_modules/appcenter-cli
> cd appcenter-file-upload-client-node && npm i

npm ERR! code EACCES
npm ERR! syscall scandir
npm ERR! path /root/.npm/_logs
npm ERR! errno -13
npm ERR!
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.
npm ERR!
npm ERR! To permanently fix this problem, please run:
npm ERR!   sudo chown -R 65534:0 "/root/.npm"
glob error [Error: EACCES: permission denied, scandir '/root/.npm/_logs'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'scandir',
  path: '/root/.npm/_logs'
}
npm ERR! code EACCES
npm ERR! syscall open
npm ERR! path /root/.npm/_cacache/index-v5/ce/ae/fd93e677a614f57b325ca5f22842c3e00f2071dd70d44f24a4b413d6fe52
npm ERR! errno -13
npm ERR!
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.
npm ERR!
npm ERR! To permanently fix this problem, please run:
npm ERR!   sudo chown -R 65534:0 "/root/.npm"
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 243
npm ERR<EMAIL_ADDRESS>preinstall: `cd appcenter-file-upload-client-node && npm i`
npm ERR! Exit status 243
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>preinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-07-31T09_44_16_062Z-debug.log

I actually have no error when installing 2.6.3
Thanks

Same for me. The new version seems corrupted.

+1

same for me +1

for anyone who struggles and don't know how to get it done:
npm cache clean --force
npm install -g appcenter-cli --force

or just use old version for now
npm install -g<EMAIL_ADDRESS>

Hi,
We're really sorry this is happening for you. Considering this mentions /root I take it this error happened on some Linux distribution? Can we have more details on the environment?

linux distribution & version?
npm version (I see 6.14.6)?
using nvm or similar npm version management tool?
previous version of appcenter-cli that was installed
using sudo? running as root or normal user?

The following message makes it look like a bug in npm and not the package, but we need to make sure.
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.

Thanks in advance

Same here. Solved by installing previous version for now npm install -g<EMAIL_ADDRESS>

Hi,
We're really sorry this is happening for you. Considering this mentions /root I take it this error happened on some Linux distribution? Can we have more details on the environment?

linux distribution & version?
npm version (I see 6.14.6)?
using nvm or similar npm version management tool?
previous version of appcenter-cli that was installed
using sudo? running as root or normal user?

The following message makes it look like a bug in npm and not the package, but we need to make sure.
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.

Thanks in advance

I am using azure pipelines with these environments:
https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-README.md
https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1804-README.md
https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md

Hi,
We're really sorry this is happening for you. Considering this mentions /root I take it this error happened on some Linux distribution? Can we have more details on the environment?
* linux distribution & version?

* npm version (I see 6.14.6)?

* using nvm or similar npm version management tool?

* previous version of `appcenter-cli` that was installed

* using sudo? running as root or normal user?

The following message makes it look like a bug in npm and not the package, but we need to make sure.
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.

Thanks in advance

I'm using it in linux docker image
https://github.com/okalman/AppCenter-Github-Action/blob/master/Dockerfile

This version started breaking our CI Pipeline.
Using Gitlab CI with image openjdk:8-jdk.
Example steps:
    - apt-get install -y curl && curl -sL https://deb.nodesource.com/setup_lts.x | bash - && apt-get install -y nodejs && curl -L https://www.npmjs.com/install.sh | sh
    - npm install -g appcenter-cli

This is happening to me too, running on a Docker image in Bamboo
Docker image: node:10-buster (Debian 10.5)
NPM version: 6.14.6
Using NVM: No
Previous version of appcenter-cli: None
Using sudo: no

Hi,
We're really sorry this is happening for you. Considering this mentions /root I take it this error happened on some Linux distribution? Can we have more details on the environment?

linux distribution & version?
npm version (I see 6.14.6)?
using nvm or similar npm version management tool?
previous version of appcenter-cli that was installed
using sudo? running as root or normal user?

The following message makes it look like a bug in npm and not the package, but we need to make sure.
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.

Thanks in advance

Hi, I am having the issue using this docker image node:12.18-alpine
No matter if I use sudo or not to run the npm install command

Okay I know what's up here.
Our preinstall script installs the locally bundled appcenter-file-upload-client-node dependency, and this runs npm i within npm install. On linux, when running as root (or sudo), npm install creates a cache and log files under /root/.npm, but there can only be one at a time. So, the second npm i sees that a file already exists in that folder and bails out.
The solution will be to switch to appcenter-file-upload-client-node from npmjs.org since it's now been published.

This should now be fixed, through a revert. We'll up our testing before releasing the change again.

This is working again for me (on version 2.6.5) - thanks!


A bug in Knip or TypeScript (Exported variable 'X' has or is using name 'Y' from external module "Z" but cannot be named.)

https://stackoverflow.com/a/77390832/368691
The StackOverflow answer has all the context, but the main problem here is that Knip recommended something to be removed because it is not being consumed, but that broke applications that depend on this package.
I am not familiar with why TypeScript would need this type to be exported though. Might be worth tagging some folks from TypeScript team on this issue.

This has become a major issue after migrating to moduleResolution NodeNext.
https://github.com/microsoft/TypeScript/issues/47663
The only thing that so far saves us is setting:
"declaration": false,
"declarationMap": false,

at the application level.

For what it's worth, Knip has uses/sets/overrides to moduleResolution: ts.ModuleResolutionKind.NodeNext for said purpose basically since the beginning.
Need to dig into this deeper, but is this not an issue when compiling your code, but it is when running Knip?

We get this when compiling the code after applying Knip suggestions.

Right, that sounds like Knip reported some false positives, and maybe the issue is with TypeScript itself. In the linked SO comment I see you reference https://github.com/gajus/roarr/commit/5c4e8d373db96c91ee3b0cd7c543ed72bf158791, so that would be the reproduction I could look into?

If you try version https://github.com/gajus/roarr/releases/tag/v7.18.0 that should help to repro the issue.
If you cannot, let me know and I will setup a repro.

Sounds like we all agree that something should not be exported. But not exporting it breaks other packages.
I guess the ask is whether Knip should (not) report it. But at this point, if I understand correctly, I'd rather see a fix on the TS end.

I guess the ask is whether Knip should (not) report it. But at this point, if I understand correctly, I'd rather see a fix on the TS end.

I am honestly completely lost about this one. I cannot even figure out how to fix it sometimes because TypeScript errors do not identify which types need to be exported.
So far I found though that it is usually the interfaces that are causing the problem and sometimes simply changing interface to a type fixes the issue. Slonik issue I linked though shows thats not always the case.

Cleaning up the issue list a bit, so I'm going to close this issue.
Recent versions may have improved the situation a bit, but since we think it's a TypeScript issue, there's probably no solution yet?
You may want to try the --isolate-workspaces option I've introduced recently to enforce isolating the workspaces so they each use their own tsconfig.json settings (Knip shares them whenever possible to improve performance and reduce memory usage).
You also may want to try v4 canary, since that version refactors the way imports and exports are matched against each other. I expect in this case it won't help much.
Not sure how to repro the issue, so leaving it at this for now.

For what it's worth, this may be fixed in TS v5.5:
https://devblogs.microsoft.com/typescript/announcing-typescript-5-5-beta/#consulting-packagejson-dependencies-for-declaration-file-generation


Creating a StateComparator Measurement Filter crashes ExpSettingsGUI

Steps to reproduce:
-Create a StateComparator in the Measurements tab
-The Comparator form is empty and the GUI crashes ( see example below)
Expected behavior:
-Form is displayed and the GUI does not crash
example of failure:

fixed in https://github.com/BBN-Q/PyQLab/commit/84cbf2342258fc3c45d4530a749eb8c943f656fd


vulcan-events: TypeError: Cannot redefine property: name

handling error for Object.defineProperty()

I still feel like that doesn't really solve the issue? Is there no way to fix the root cause instead of just catching the error?

Closing as stale


contact you re mistake on ppdcl website

Hi, I don't know how to get to your email address. So I use this page.
Thanks for freely creating the website ppcdl.sn.sg to benefit fellow test takers. I failed once. Ha ha. Like to point out the error on 1st set of questions, Q 21.
When proceeding west, if you see a east cardinal buoy, you should :
the correct answer that is correct is: u-turn and go back. It should be "alter course 45 degrees to port or starboard and proceed till the buoy is due north or south. "
Thanks for your generosity. kim ong.
<EMAIL_ADDRESS>Somehow I can't get twitter to work.

Hi Kim,
Thanks for your kind words and for taking time to correct my mistake! I have applied the fix immediately.
I'll also email you a copy of this comment and remove this issue.
Have a good holiday!


MongoDB outdated and doesnt work with RocketChat.

Last night I tried doing a clean install using just the tools in this script. I found that RocketChat would not work with the version on MongoDB provided here. I had to follow the instructions at http://andyfelong.com/downloads/core_mongodb.tar.gz for 3.0.9 and then use DPlatform to install rocketchat.

We was using its build before a mongodb build was available in the Debian Stretch repository.
Maybe it's just the name of the service, I remember an issue like this, mongodb.service instead of mongod.service.

Should fix the issue.
By the way RC worked before, but the DB just not started wirh the it right?


ntr curved elongated vegetative cell with long curved microtubules PMID:11739790

PMID:11739790
curved elongated vegetative cell with long curved microtubules
descendent of
FYPO:0005103	curved elongated vegetative cell
FYPO:0004863	long curved microtubules
parent of
FYPO:0004864	viable curved vegetative cell with long curved microtubules

Ideally, I'd also like
curved "J" shaped elongated vegetative cell with long curved microtubules
curved "C" shaped elongated vegetative cell with long curved microtubules
(the one with G2 block has the bend in the middle,  G1 block ha bend at the end (elongation is only from one end)

hmm , the curved microtubules here aren't just curved followin the cell, they are 'proper looped around end of cell"

This bit

parent of
FYPO:0004864 viable curved vegetative cell with long curved microtubules

would cause a TPV because FYPO:0004864 does not specify that the cell itself is elongated.
Looped vs not-looped MTs will have to go just in definition text.
(note to self: can request 'C-shaped' and 'J-shaped' from PATO; looks like there's ample precedent for other letters)

Could we word it in some way so that the "elongated microtubules"  curve around the cell end. The normal length ones stop at the cell tip.
So  in H above it would be
elongated cell with elongated microtubules (curling around the cell tip can be in the def)
But F would be normal length microtubules, because they are normal for a long cell. They still stop at the cell tip.
Does that make any sense?

opened https://github.com/pato-ontology/pato/issues/96

curved elongated vegetative cell with long curved microtubules FYPO:0005425
elongated C-shaped vegetative cell with long curved microtubules FYPO:0005426
elongated J-shaped vegetative cell with long curved microtubules FYPO:0005427

edit file: a760fa43a796d2330cd62ac432969cc41ec40ced
release: b51f1da33aae79745f62d7c2015f90d7a38c922a


How can i translate Speech to Text?

I have Create Application to translate text-to-text , text-to-speech.
Now I want to translate Speech-to-speech and Speech-to-text.
How can i use your demo to translate Speech-to-text and Speech-to-Speech ?
How can i send audio in chunks to microsoft API?
In Objective-C
Please Help me.

Sorry, this is a pure text-text API only.


feat: expose no_install and install_only as public API

Just an idea, but I feel it convenient if they are exposed as public API.
I'm using pip-sync to install dependencies, so I want to re-use --no-install flag for disabling running pip-sync. I sometimes pass posargs to external tools like pytest, so I want to avoid nox -s ~ -- --no-sync.
I feel that some other people may find them useful, but I don't have any other use case. Suggestions are welcome 🙂

I often use -r and --no-install simultaneously to speed up tests. Also, my point is to keep the behavior of session.install and session.run("pip-sync") consistent when --no-install or --install-only is given.

I often use -r and --no-install simultaneously to speed up tests.

In case you didn't know, you can use -R as a shortcut for this combo (see here)
What's your workflow where you need this feature? Could you give me a little example of what you'd do with these new attributes I cant quite picture why you'd need it in my head

@FollowTheProcess
Like this snippet:
if not session.no_install or session.install_only:
    session.run("pip-sync", requirements)

Ahh okay that makes sense, in that case I'm happy with this. Just need to get these covered in a test to make the CI green and we should be okay 👍🏻

Thanks, I just added some tests to make coverage happy

This looks like the use case for session.run_always, or am I missing something? That function won't run under --no-install.

This looks like the use case for session.run_always, or am I missing something? That function won't run under --no-install.

Ah, you're right, thank you. I feel like it'd be a better solution than manually writing if sentences. I still think that it's OK to expose these variables since they are specified by users, but now I don't have strong motivation to merge this PR 😅

Ahh nice! is this still required then @kngwyu ?

I'm sorry but I don't have clear motivation now


crash on manual push job execution when the sink dataset is not created yet

I'm testing a simple manual push job to sync to external drive and I'm hitting a crash.
The crash stack is following:
❯ zrepl --config zrepl.yml daemon
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xaefc8f]

goroutine 194 [running]:
github.com/dsh2dsh/zrepl/replication/logic.(*Filesystem).doPlanning(0x86c14e070, {0xfbb000, 0x86c11ea20}, 0x1)
        github.com/dsh2dsh/zrepl/replication/logic/replication_logic.go:530 +0xcef
github.com/dsh2dsh/zrepl/replication/logic.(*Filesystem).PlanFS(0x86c1223d0?, {0xfbb000?, 0x86c11ea20?}, 0x0?)
        github.com/dsh2dsh/zrepl/replication/logic/replication_logic.go:141 +0x25
github.com/dsh2dsh/zrepl/replication/driver.(*fs).do.func1()
        github.com/dsh2dsh/zrepl/replication/driver/replication_driver.go:536 +0x129
github.com/dsh2dsh/zrepl/util/chainlock.(*L).DropWhile(0x2bea895d7ae8?, 0x86c017ba0?)
        github.com/dsh2dsh/zrepl/util/chainlock/chainlock.go:40 +0x70
github.com/dsh2dsh/zrepl/replication/driver.(*fs).do(0x86c0b4320, {0xfbb000, 0x86c11ea20}, 0x86c1223d0, 0x0, 0x1, {0xfbb038, 0x86c09fb30})
        github.com/dsh2dsh/zrepl/replication/driver/replication_driver.go:525 +0x255
github.com/dsh2dsh/zrepl/replication/driver.(*attempt).doFilesystems.func1(0x86c0b4320)
        github.com/dsh2dsh/zrepl/replication/driver/replication_driver.go:480 +0x138
created by github.com/dsh2dsh/zrepl/replication/driver.(*attempt).doFilesystems in goroutine 117
        github.com/dsh2dsh/zrepl/replication/driver/replication_driver.go:474 +0x219

I have cleaned the zrepl.yml confing as much as possible.
jobs:
  - name: "hourly-snap"
    type: "snap"
    filesystems: &filesystems
      "mpool/test": true
    snapshotting:
      type: "periodic"
      cron: "10 * * * *"
      prefix: "zrepl_hourly_"
    pruning: &keep-all

  - name: "bpool"
    type: "sink"
    root_fs: "bpool/zrepl"
    serve:
      type: "local"
      listener_name: "bpool"

  - name: "mpool-to-bpool"
    type: "push"
    connect:
      type: "local"
      listener_name: "bpool"
      client_identity: "localhost"
    filesystems: *filesystems
    snapshotting:
      type: "manual"
    pruning:
      keep_sender:
        - type: "regex"
          negate: true
          regex: "^zrepl_"
        - type: "last_n"
          count: 2
          regex: "^zrepl_hourly_"
      keep_receiver:
        - type: "last_n"
          count: 90
          regex: "^zrepl_hourly_"

the mpool/test is a simple empty dataset with one zrepl_hourly_xxx snapshot. The bpool/zrepl is newly created dataset without any descendants.
It crashes when I call
zrepl signal wakeup mpool-to-bpool
This is in the log:
2024/10/18 18:14:34 INFO got signal subsystem=internal job=_control op=wakeup
2024/10/18 18:14:34 INFO with RPC timeout subsystem=rpc.ctrl job=mpool-to-bpool timeout=1m0s
2024/10/18 18:14:34 INFO start replication subsystem=job job=mpool-to-bpool
2024/10/18 18:14:34 INFO start planning subsystem=repl job=mpool-to-bpool
2024/10/18 18:14:34 INFO "/sbin/zfs list -H -p -o name -r -t filesystem,volume" subsystem=zfs.cmd job=mpool-to-bpool
2024/10/18 18:14:34 INFO "/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl" subsystem=zfs.cmd job=bpool
2024/10/18 18:14:34 INFO "/sbin/zfs list -H -p -o name -r -t filesystem,volume" subsystem=zfs.cmd job=bpool
2024/10/18 18:14:34 INFO "/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder mpool/test" subsystem=zfs.cmd job=mpool-to-bpool
2024/10/18 18:14:34 INFO "/sbin/zfs list -H -p -o name,guid,createtxg,creation,userrefs -r -d 1 -t bookmark,snapshot -s createtxg mpool/test" subsystem=zfs.cmd job=mpool-to-bpool
2024/10/18 18:14:34 INFO conflict automatically resolved subsystem=repl job=mpool-to-bpool conflict="no common snapshot or suitable bookmark between sender and receiver:\n  sorted sender versions:\n    @zrepl_hourly_xxx\n  sorted receiver versions:\n" filesystem=mpool/test

This is on a current freebsd 14.1 zrepl-dsh2dsh version 0.8.5
I have also tested a slightly modified config that works also on original zrepl and with the original zrepl it works ok. If I run this on the zrepl-dsh2dsh it fails the same.
jobs:
  - name: "hourly-snap"
    type: "snap"
    filesystems: &filesystems
      "mpool/test": true
    snapshotting:
      type: "cron"
      cron: "10 * * * *"
      prefix: "zrepl_hourly_"
    pruning:
      keep:
        - type: "regex"
          regex: ".*"

  - name: "bpool"
    type: "sink"
    root_fs: "bpool/zrepl"
    serve:
      type: "local"
      listener_name: "bpool"
    recv:
      placeholder:
        encryption: inherit

  - name: "mpool-to-bpool"
    type: "push"
    connect:
      type: "local"
      listener_name: "bpool"
      client_identity: "localhost"
    filesystems: *filesystems
    snapshotting:
      type: "manual"
    pruning:
      keep_sender:
        - type: "regex"
          negate: true
          regex: "^zrepl_"
        - type: "last_n"
          count: 2
          regex: "^zrepl_hourly_"
      keep_receiver:
        - type: "last_n"
          count: 90
          regex: "^zrepl_hourly_"

I have not tested the master branch version of zrepl yet.

Thanks for the report. I'll take a look.

I'm testing a simple manual push job to sync to external drive and I'm hitting a crash.

Please download latest release
https://github.com/dsh2dsh/zrepl/releases/tag/v0.8.7
and try again. I hope I fixed it.

The crash seems to be fixed, but still has an issue with initial replication.
❯ ./zrepl --config zrepl.yml daemon
2024/10/18 21:38:01 ERROR command exited with error subsystem=zfs.cmd job=mpool-to-bpool cmd="zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost" err="wait \"/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost\": exit status 1" status=1 systemtime_s=0 total_time_s=0.003441288 usertime_s=0.003835
2024/10/18 21:38:01 ERROR receive request failed (might also be error on sender) subsystem=repl job=mpool-to-bpool err="cannot get placeholder state of bpool/zrepl/localhost: zfs exited with error: wait \"/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost\": exit status 1\nstderr:\n" errType=*fmt.wrapError filesystem=mpool/test rr="Filesystem:\"mpool/test\" To:{Name:\"zrepl_hourly_xxx\" Guid:11229806340946721345 CreateTXG:36905810 Creation:\"2024-10-18T18:09:21+02:00\"} ClearResumeToken:true ReplicationConfig:{protection:{Initial:GuaranteeResumability Incremental:GuaranteeResumability}}"
2024/10/18 21:38:01 ERROR most recent error in this attempt subsystem=repl job=mpool-to-bpool attempt_number=0 err="cannot get placeholder state of bpool/zrepl/localhost: zfs exited with error: wait \"/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost\": exit status 1\nstderr:\n"
2024/10/18 21:38:01 ERROR most recent error cannot be solved by reconnecting, aborting run subsystem=repl job=mpool-to-bpool attempt_number=0

And thanks for a very quick response. I really appreciate the integration of one step stream send. This is something why I get rid of zrepl time ago and replaced it with a simple script to sync only daily snapshots.

The crash seems to be fixed, but still has an issue with initial replication.

Looks like bpool/zrepl/localhost doesn't exists. Did you create it? In pkg-message I'm suggesting
- Create ZFS datasets like:
  # zfs create -o mountpoint=none -o readonly=on zdisk/zrepl
  # zfs create zdisk/zrepl/localhost

created, or not, the error is similar only level lower
2024/10/18 21:47:57 ERROR command exited with error subsystem=zfs.cmd job=mpool-to-bpool cmd="zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost/mpool" err="wait \"/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost/mpool\": exit status 1" status=1 systemtime_s=0.004059 total_time_s=0.003618258 usertime_s=0
2024/10/18 21:47:57 ERROR receive request failed (might also be error on sender) subsystem=repl job=mpool-to-bpool err="cannot get placeholder state of bpool/zrepl/localhost/mpool: zfs exited with error: wait \"/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost/mpool\": exit status 1\nstderr:\n" errType=*fmt.wrapError filesystem=mpool/test rr="Filesystem:\"mpool/test\" To:{Name:\"zrepl_hourly_xxx\" Guid:11229806340946721345 CreateTXG:36905810 Creation:\"2024-10-18T18:09:21+02:00\"} ClearResumeToken:true ReplicationConfig:{protection:{Initial:GuaranteeResumability Incremental:GuaranteeResumability}}"
2024/10/18 21:47:57 ERROR most recent error in this attempt subsystem=repl job=mpool-to-bpool attempt_number=0 err="cannot get placeholder state of bpool/zrepl/localhost/mpool: zfs exited with error: wait \"/sbin/zfs get -Hp -o name,property,value,source zrepl:placeholder bpool/zrepl/localhost/mpool\": exit status 1\nstderr:\n"
2024/10/18 21:47:57 ERROR most recent error cannot be solved by reconnecting, aborting run subsystem=repl job=mpool-to-bpool attempt_number=0

Even if I create bpool/zrepl/localhost/mpool, it still fail to create the last test

created, or not, the error is similar only level lower

Another fix. Please destroy bpool/zrepl/localhost/mpool and bpool/zrepl/localhost and try again.
Are you able to build zrepl from master branch or would you like to build it for you?

Thanks, it seems to be working fine now.
The errors I mentioned remains, but I think it is expected.
I have tested that on more datasets in my pool and all of them replicated just fine.

The errors I mentioned remains, but I think it is expected.

Could you show me log with remained errors?


Enable Public CORS Policy on all Controllers

i.e. for projects
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://c4sg.herokuapp.com/api/projects/all. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

Hi,
I do not get this issue in my local. Getting the expected response and the code has the @CrossOrigin annotated on every endPoint already.

@arpitsrm, You are correct. CrossOrigin should have been taken care of at backend, so that localhost:4200 could communicate successfully with localhost:8080. I assume @russon77 is looking into frontend on how to address the error, he might provide some hints if changes are expected at backend.

This issue was moved to Code4SocialGood/c4sg-services#7


JS-Grid right-click context menu

Hi,
I have been looking all over, but coul not find any topic to right click context menu.
There are these two:
rowClick: function(args) { ... },
rowDoubleClick: function(args) { ... },
Any chance of getting the row context mouseclick to fire.
thanx

I am not sure if this feature exists?


[fix][doc]Add docs for ThresholdShedder

Motivation
Add docs for ThresholdShedder
https://github.com/apache/pulsar/pull/17456
Documentation

[x] doc
[ ] doc-required
[ ] doc-not-needed
[ ] doc-complete

@315157973 thanks for adding the docs!
Since it works on top of ThresholdShedder, does it make sense to move this newly added content a bit later (starting from here), following the complete content of ThresholdShedder?

@315157973 thanks for adding the docs! Since it works on top of ThresholdShedder, does it make sense to move this newly added content a bit later (starting from Row168), following the complete content of ThresholdShedder?

This part explain the load balancing strategy. If we divide them into two parts, the meaning will be incoherent

@315157973 thanks for adding the docs! Since it works on top of ThresholdShedder, does it make sense to move this newly added content a bit later (starting from Row168), following the complete content of ThresholdShedder?

This part explain the load balancing strategy. If we divide them into two parts, the meaning will be incoherent

Also make sense. My thought - it breaks the coherent content of ThresholdShedder and also has a dependency on the configuration of ThresholdShedder, so it looks a bit advancing if we introduce it ahead of ThresholdShedder. It's not a broker anyway :)


[1 of 3]: Create FluidGeminiContainer component

This PR introduces a wrapper component for Gemini which essentially does one thing: creates CSS to override Gemini's inline width property. It also renames the GeminiUtil to ScrollbarUtil and adds a method to get the width of the browser's scrollbar.
This is necessary when the width of the container of a Gemini element is animated. A good example of is the Page's width, when you toggle between a docked and undocked sidebar.
Without this change, the width of Gemini's view container is set to a value in pixels, and while the container's width animates, the native scrollbar is visible. With this change, the width of Gemini's view container will always match its container's width, plus an offset equal to the width of the browser's scrollbar.
If you'd like to test this, replace GeminiScrollbar with FluidGeminiScrollbar in Page.js and toggle the sidebar's docked state. You'll see the width of Page's children animate along with the Sidebar.

Pushed a few changes @jfurrow

Thanks @rcorral! I just pushed another small change (I didn't set the . styleSheet.cssText property correctly, though it seems to work either way).

I'm ok with this component but wonder if it is actually worth all the effort, simply to have a "better" looking scrollbar. Every once in a while, we have serious issues with the GeminiScrollbar like the form state loss, unwanted updates, wrong dimensions, and visual issues like the one you're fixing.
Let's discuss this with design team before we continue to fix the GeminiScrollbar as I don't want us to waste time with something that cost so much but brings so little. It's tiny and decreasing number of users that is still doomed to see ugly scrollbars.
/cc @dcos/design @dcos/frontend

@orlandohohmeier This is a good point. Right now I am aware of at least one fatal bug with the scrollbar on Ubuntu (both FF and Chrome) that still needs to be fixed.


Parsejson library Version

As reported by Github:
Known high severity security vulnerability detected in parsejson <=0.0.3 defined in package-lock.json.

npm audit showed the issue to be in parsejson. Then I tried to fix the issue with npm install parsejson@latest but this is a know issue:
https://www.npmjs.com/advisories/528
It seems that we need to change the code to use JSON.parse() from Node.js


if wpa peap username or password is not configured, build nak response message and send back to authenticator (IDFGH-6088)

Fixed the issue that unable to use Wpa Tls to connect to windows radius server when the radius serer is configured to check PEAP and then TLS.
If radius server's rule is configured to authenticate PEAP and then TLS and esp-idf is configured as Tls, esp-idf needs to respond a nak message to radius server to trigger to it to retry with Tls.

@sagb2015 to validate config within eap_peer_get_eap_method(...)? the check should be done within 'peap'-specific function instead of in generic logic. I noticed eap_tls_init(...) checking config->private_key, that's why my proposed change is checking username and password in eap_peap_init(...).

@sagb2015 could you give more advise on this PR?

Hi @feifranklin, can you help to test with this patch ? If it works okay, we will soon have it available in master.

Hi @feifranklin, can you help to test with this patch ? If it works okay, we will soon have it available in master.

@feifranklin There are certain problems with this patch as well. We are working on the better solution,

Hi @feifranklin, can you help to test with this patch ? If it works okay, we will soon have it available in master.

@feifranklin There are certain problems with this patch as well. We are working on the better solution,

@sagb2015  keep me posted. I can help testing once the fix is available.

@feifranklin Your method works, but ideally if PEAP is not configured, the corresponding init itself should not be called. In other words, the check should happen even before init is called IMO. The configuration should be enough to determine what the device can support. The problem stems from the fact traditionally ESP devices aimed to support all the methods, making all the config mandatory. However, later on, some part of config was made optional assuming not all the methods will be needed in the field. This now means config should derive supported methods and only proceed with those while handling EAP messages. Hope this is clear!

@feifranklin Can you please tryout the following patch and let us know how it behaves and if the NAK message goes through.
eap_method_selection_patch.txt

Thanks for your contribution, changes merged with https://github.com/espressif/esp-idf/commit/190b31bb1b9b1e6e21feb038797aed6e14cb765b.


Improve .card-img-overlay

Currently, the .card-img-overlay only properly works for cards that use the img as their full background, because the overlay will span the whole .card. If the .card has content below the image, then the .card-img-overlay can overflow the image, and it's not possible to align elements to the bottom of the image. For images that are not the first element of a card the overlay breaks completely.
This is very simply solved with a wrapper div around .card-img and .card-img-overlay with position: relative;. JS Bin for comparison: https://jsbin.com/hixuvegihu/edit?html,output
So a single class like .card-img-overlay-wrapper would solve this issue and make the img-overlay much more flexible to use.

Closing as a dupe of #18476 which asks for a position: relative utility that'd help you out here.


setState mutates state instead of replacing it

Currently in Preact 8.2.9, Component.prototype.setState mutates the previous state instead of replacing it with a patched version. This is unnecessarily inconsistent with React, can break user intuitions, can break things ported from React, and makes the second argument to shouldComponentUpdate useless and misleading.
I discovered this the hard way when sharing an initial state constant between components, assuming it would be treated as immutable. As previously mentioned, this can break user intuitions when coming from React.
Can we fix this? I would be willing to do a PR.

Definitely definitely happy to see a PR for this if you have the time. In the meantime do you have a concise test we could use to reproduce the issue?

Here's a simple reproduction (run in Node):
'use strict'

const preact = require('preact')
const render = require('preact-render-to-string')

const STATE = {}

class Component extends preact.Component {
  constructor() {
    super(...arguments)
    this.state = STATE
  }

  componentWillMount() {
    this.setState({key: 'value'})
  }
}

console.info(STATE)
// {}

render(preact.h(Component))

console.info(STATE)
// { key: 'value' }

// Expected behavior: STATE remains {}

Hello I've fixed this bug but have never contributed before. I'm willing to send a PR or describe the fix for someone else. I'm a bit hesitant since I'm new to the codebase. I have a test and two modifications.
	it("issue 1168, state should not be mutated", () => {
		const STATE = {};
		class C1168 extends Component {
			constructor() {
				super(...arguments);
				this.state = STATE;
			}
			componentWillMount() {
				this.setState({key: 'value'});
			}
		}
		render(<C1168 />, scratch);
		expect(STATE).to.deep.equal({});
	});

src/component.js Component.prototype:
	setState(state, callback) {
		if (!this.prevState) this.prevState = this.state;
		this.state = extend(extend({},this.state), typeof state==='function' ? state(this.state, this.props) : state);
		if (callback) this._renderCallbacks.push(callback);
		enqueueRender(this);
	},

src/vdom/component.js renderComponent:
	if (component.constructor.getDerivedStateFromProps) {
		state = extend({},state);
		component.state = extend(state, component.constructor.getDerivedStateFromProps(props, state));
	}

Bundle size decreases by three bytes, from 3494 to 3491, and there is not a noticeable performance difference. I did not, however, check if this has an effect on issue 1169, but will shortly.

Oh I guess someone beat me to it, with issue 1170.

This should be fixed in 8.3.0!
Please feel free to re-open otherwise :)


Implement REST API for IoTDB

Implement REST API for IoTDB

By the way, please use spotless:apply~


Uplinks don't work

Description

Title explains itself, uplinks don't work.  You also get some error in the console about how a sound couldn't be played.

Fixed in #9765
this also only applies to handheld uplinks, not pda syndie ones.


Implement hit counter in the cucumber-runner

Implement the hit counter for Cucumber 🥒. This is to prevent infinite loops from resulting in a timeout, see #3023.

Yeah, it's all pretty much the same. Kind of makes you think to create a base class for plane node runners

This feature was released in StrykerJS 5.5, see https://twitter.com/stryker_mutator/status/1463193766982995978


ILS Workflow: Export to Rapid ILS for electronic serials monthly

See https://github.com/TAMULib/fw-registry/issues/105 for details.
Rapid Electronic Serials
There is a job on killick.library.tamu.edu, /exlibris/sfx_ver/sfx4_1/tamu/home/scripts/rapid_sfx.pl. It takes an existing file that is created by an SFX utility and copies it to a folder on killick from which it can be downloaded. The replacement job needs to follow the same procedure to copy the output file to the MIS dashboard with the filename rapid_electronic_serials.gz The file should also be gzipped, which the current script doesn't do. Cron job should be run monthly on the 10th of the month.

Where will the SFX utility be run from and where will it store the file. Will this require an FTP copy of the SFX utility output file?

#!/exlibris/sfx_ver/sfx4_1/app/perl/bin/perl

use strict;
use warnings;
use lib qw(/home/exlibris/perl/lib);
use TAMU_General;
use Net::FTP;
use File::Copy;

##############################################################################
# Variables
##############################################################################
my $institute = 'TAMUCS';
my $contact =<EMAIL_ADDRESS>##############################################################################
my $file_dir = '/exlibris/sfx_ver/sfx4_1/tamu/dbs/scratch';
my @files = glob("$file_dir/e-collection-$institute.*");
my $outfile = pop(@files);
copy("$outfile", "/home/hismith/rapid_electronic_serials");
system ("echo 'Upload /home/hismith/Journal_Electronic to https://rapid.exlibrisgroup.com/holdings'|mailx -s 'Monthly SFX upload to Rapid' $contact");


fix ProducerPerformance bug when numRecords > Integer.MAX, it will fa…

…ll into non-stop loop and send more message to broker, and Stat in PerfCallback method record will throw ArrayIndexOutOfBoundsException
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

how to


complete base pdf extractor

Extract Text from pdf, return as list of pages

View the Script Output
360 is "\n" because the uploaded file is a scan, not an actual document. Some syllabi are far from perfect. The majority seem fine.

View the Script Output
360 is "\n" because the uploaded file is a scan, not an actual document. Some syllabi are far from perfect. The majority seem fine.

360 is "\n" because the uploaded file is a scan

What do you mean by scan and not document?

360 is "\n" because the uploaded file is a scan

What do you mean by scan and not document?

360 is "\n" because the uploaded file is a scan

What do you mean by scan and not document?

It's a photoscan of the pdf. It contains no text data in the file.

360 is "\n" because the uploaded file is a scan

What do you mean by scan and not document?

It's a photoscan of the pdf. It contains no text data in the file.

great start lgtm thanks!!

Thanks for the review. Merging now. NLP stuff will happen in another branch.

great start lgtm thanks!!

Thanks for the review. Merging now. NLP stuff will happen in another branch.

It's a photoscan of the pdf. It contains no text data in the file.

Are you counting on all of the syllabi being "True PDFs"? Or are you gonna be doing OCR stuff on the ones that aren't
If not, what should we do with the syllabi that are image-only?

It's a photoscan of the pdf. It contains no text data in the file.

Are you counting on all of the syllabi being "True PDFs"? Or are you gonna be doing OCR stuff on the ones that aren't
If not, what should we do with the syllabi that are image-only?

Also,

Some syllabi are far from perfect.

Can you make a list of them and send it over? Maybe we can fix them with some online OCR tools like Sandwich PDF (that's supposedly an alternative name for PDFs that are made searchable lmfao), or ask the professors for a proper copy.

Also,

Some syllabi are far from perfect.

Can you make a list of them and send it over? Maybe we can fix them with some online OCR tools like Sandwich PDF (that's supposedly an alternative name for PDFs that are made searchable lmfao), or ask the professors for a proper copy.

The output of this script and its successor can be found here. Feel free to edit the original repo (the source of our content). I think on our part (e.g. on our site), we'll have a good way to mark missing or garbled syllabi, so certainly no urgency on that front. Something like 95+ percent accuracy is good enough for MVP status.

The output of this script and its successor can be found here. Feel free to edit the original repo (the source of our content). I think on our part (e.g. on our site), we'll have a good way to mark missing or garbled syllabi, so certainly no urgency on that front. Something like 95+ percent accuracy is good enough for MVP status.


IMAP wrong message sorting

This issue was mentioned already on the old Google Code issue tracker here:
https://code.google.com/p/k9mail/issues/detail?id=726
Unfortunately it is obviously still not fixed as I see the same issue with wrong message sorting when using IMAP also with the latest version.
Although I really like K9-Mail due to its clean interface this makes the program completely unusable for me.
Is there any chance that this bug gets fixed?

Sounds like you're sorting messages by "Arrival". Change the sorting in the message list screen to "Date" and you should get the behavior you want.

That indeed did the trick - thanks a lot for your quick reply.
I really could have found out this myself. ;-)
Thanks again
Andreas


Fix handling of .override in attr_* chains at end of signature

Added support for the .override modifier in method signatures when it appears at the end of the chain for attr_reader, attr_writer, and attr_accessor. Previously, the rewriter only handled .override correctly when it appeared at the beginning of the chain.
This was done by modifying Util.cc to ensure that .override is properly handled during signature chain traversal by adding a condition to identify it explicitly.
Motivation
Fixes #4456: "Rewriter doesn't parse sigs with override"
Test plan
Added a regression test in test/testdata/rewriter/attr_override.rb that verifies:

.override works in different positions in the signature chain
Combinations with other modifiers (like .checked) work properly

Locally tested with the UBSan and ASan config.

It would have been nice to take the opportunity to fix this castSig function to make it work with overridable as well, and possibly even with abstract though I don't think there are any uses of castSig where it would be valid to combine with abstract at the moment (probably would be good to fix it on the off chance that we add a new codepath using this method at some point in the future).
e.g., The same bug in #4456 reproduces with overridable

It would have been nice to take the opportunity to fix this castSig function to make it work with overridable as well, and possibly even with abstract though I don't think there are any uses of castSig where it would be valid to combine with abstract at the moment (probably would be good to fix it on the off chance that we add a new codepath using this method at some point in the future).
e.g., The same bug in #4456 reproduces with overridable

I'll take a look at this and see if it's something I can do.


moves ansible role to CH repo

replaces christiangda/ansible-role-amazon-cloudwatch-agent in companieshouse/amzn2-base-ami/ansible/requirements.yml

Needs testing


Regeneration effect not applying correctly.

The regeneration effect gives the visual effect of it working while wearing the set, but won't actually give regeneration until I remove a piece of the set for some reason. I have just been editing the example.
val iron_set = new ArmorSetEffect("iron_set")                .inSlot(<equipmentslottype:head>, <item:minecraft:iron_helmet>)                .inSlot(<equipmentslottype:chest>, <item:minecraft:iron_chestplate>)                .inSlot(<equipmentslottype:legs>, <item:minecraft:iron_leggings>)                .inSlot(<equipmentslottype:feet>, <item:minecraft:iron_boots>)                .setIgnoreNBT()                .addEffect(<effect:minecraft:regeneration>.newInstance(50, 3))                .addImmunity(<effect:minecraft:blindness>)                .addAttackerEffect(<effect:minecraft:poison>.newInstance(60, 1)); iron_set.register();

I just read the other open issue's further comments and can work with the timer effect issue for now, thanks!

Yep! Sorry about that, Regeneration, Hunger, Poison and Wither are modified according to their duration in vanilla.


Subsystem Reset issue ERROR nvme_ctrlr.c

Hi crane
I tested pynvme at 2 platform (1 desktop, 1 server)
but both system can not operate subsystem reset
pynvme occur error below
--desktop log--
ERROR nvme_ctrlr.c(728) nvme_ctrlr_shutdown: get_csys() failed
python3: nvme_ctrl.c:728: nvme_ctrlr_shutdown: Assertion 'false' failed
--server log--
INFO Driver.c(498) probe_cb: Attaching to NVMe Controller at 0000:xx:xx.x
ERROR nvme_pcie.c(488) nvme_pcie_ctrlr_map_cmb: get registers failed
python3: nvme_pcie.c:488: nvme_pcie_ctrlr_map_cmb: Assertion 'false' failed.
any sample can not subsystem reset.
i use master brance

From NVMe Spec:

10.5 Controller Fatal Status Condition
Performing an NVM Subsystem Reset (NSSR) may cause PCI Express links to go down as part of resetting the NVM subsystem. Host software may have undesirable effects related to PCI Express links going down (e.g., some host operating systems or hypervisors may crash).

What you observed is just because PCIe links down.

We also fill confuse about this operation. Some drives fail as the same link down issue, but some drives do pass. Need more investigation.

Thank you
i will try more sample and system
availble pcie link setup in driver level?

availble pcie link setup in driver level?

You can try Pcie.reset()