[TASK] Allow users to specify "case-info" constants

Issue Description
Users should be able to provide information pertaining to a case that would be valuable to have at the end of the RAVEN stack. This could be information that would help distinguish between runs when the data contains a multilevel structure.
For example, If a user is examining effect on marginal costs across a set of varying market demand scenarios for a set of different markets, a user should be able to specify something like:
<Case name="Sweep_Runs">
...
  <caseinfo>
    <state>California</state>
    <scenario>CarbonTax</scenario>
  </caseinfo>
...
</Case>

This would allow HERON's template builder to handle a broader range of cases.

For Change Control Board: Issue Review
This review should occur before any development is performed as a response to this issue.

[ ] 1. Is it tagged with a type: defect or task?
[ ] 2. Is it tagged with a priority: critical, normal or minor?
[ ] 3. If it will impact requirements or requirements tests, is it tagged with requirements?
[ ] 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[ ] 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure
This review should occur when the issue is imminently going to be closed.

[ ] 1. If the issue is a defect, is the defect fixed?
[ ] 2. If the issue is a defect, is the defect tested for in the regression test system?  (If not explain why not.)
[ ] 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
[ ] 4. If the issue is a defect, does it impact the latest release branch? If yes, is there any issue tagged with release (create if needed)?
[ ] 5. If the issue is being closed without a pull request, has an explanation of why it is being closed been provided?

More generally, this would allow passing of constants through that aren't strictly the capacities of units; for example, buy-sell thresholds for batteries in a theoretical marginal dispatch.
This should be straightforward by adapting the workflow writer so that these are constants in the Outer sampler, passed through to the RAVEN model in the Outer, and made accessible to the DispatchManager model in the Inner.

Approved to close via #41


Error when running suricata in container

When running dpdk in docker container, I got the following error
root@bb5ca8ba6747:~/DPDK_SURICATA-4_1_1/suricata-4.1.4# suricata --dpdk=mysuricata.cfg
3/9/2019 -- 15:43:17 -  -  DPDK Mode selected
3/9/2019 -- 15:43:17 -  - This is Suricata version 4.1.4 RELEASE
*** Error in `/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata': malloc(): memory corruption: 0x0000000000f78ca0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ffff523d7e5]
Could you help suggest what is the problem

Hi @tritm,
Can you help me your environment better

Docker OS:
CPU cores:
memory assigned:
Complete logs:
step in trace from GDB core dump (build with -g)

Early Analysis:

'DPDK Mode selected' is from 'ParseCommandLine' on slecting '--dpdk'
It looks like error is from 'GlobalsInitPreConfig' where malloc has failed.
if it have suceeded we will have 'ParseDpdkYaml' with information 'Read DPDK contents YAML file'.

@tritm can you share your updates?

@tritm I am closing the ticket for now as there is no update. Please feel free to reopen once enough information is collected.

Hello @vipinpv85 I'm sorry for late reply, I re-install the system and got the following error
root@e8b92d967404:~/DPDK_SURICATA-4_1_1/suricata-4.1.4# suricata --dpdk=mysuricata.cfg
29/10/2019 -- 07:34:19 -  -  DPDK Mode selected
29/10/2019 -- 07:34:19 -  - This is Suricata version 4.1.4 RELEASE
...
EAL: FATAL: Invalid 'command line' arguments.
29/10/2019 -- 07:34:19 -  - [ERRCODE: SC_ERR_DPDK_CONFIG(319)] -  No DPDK ports found
But, I have already config dpdk port in mysuricata.cfg:
--vdev=virtio_user1,queues=1,path=/var/run/usvhost1
--vdev=virtio_user2,queues=1,path=/var/run/usvhost2
And, in the container, I have 2 mapped ports:
root@e8b92d967404:~/DPDK_SURICATA-4_1_1/suricata-4.1.4# ls /var/run
dpdk  initctl  lock  log  mount  sendsigs.omit.d  shm  systemd  user  usvhost1  usvhost2  utmp
Docker OS: Ubuntu 16.04
CPU cores: Intel(R) Xeon(R) CPU D-1548
memory assigned: 64G
Complete logs:
root@e8b92d967404:~/DPDK_SURICATA-4_1_1/suricata-4.1.4# suricata --dpdk=mysuricata.cfg
29/10/2019 -- 07:44:09 -  -  DPDK Mode selected
29/10/2019 -- 07:44:09 -  - This is Suricata version 4.1.4 RELEASE
Usage: ./dpdk-suricata [options]
EAL common options:
-c COREMASK         Hexadecimal bitmask of cores to run on
-l CORELIST         List of cores to run on
The argument format is [-c2][,c3[-c4],...]
where c1, c2, etc are core indexes between 0 and 128
--lcores COREMAP    Map lcore set to physical cpu set
The argument format is
'<lcores[<EMAIL_ADDRESS>lcores and cpus list are grouped by '(' and ')'
Within the group, '-' is used for range separator,
',' is used for single number separator.
'( )' can be omitted for single element group,
'@' can be omitted if cpus and lcores have the same value
-s SERVICE COREMASK Hexadecimal bitmask of cores to be used as service cores
--master-lcore ID   Core ID that is used as master
--mbuf-pool-ops-name Pool ops name for mbuf to use
-n CHANNELS         Number of memory channels
-m MB               Memory to allocate (see also --socket-mem)
-r RANKS            Force number of memory ranks (don't detect)
-b, --pci-blacklist Add a PCI device in black list.
Prevent EAL from using this PCI device. The argument
format is domain:bus:devid.func.
-w, --pci-whitelist Add a PCI device in white list.
Only use the specified PCI devices. The argument format
is <[domain:]bus:devid.func>. This option can be present
several times (once per device).
[NOTE: PCI whitelist cannot be used with -b option]
--vdev              Add a virtual device.
The argument format is [,key=val,...]
(ex: --vdev=net_pcap0,iface=eth2).
--iova-mode   Set IOVA mode. 'pa' for IOVA_PA
'va' for IOVA_VA
-d LIB.so|DIR       Add a driver or driver directory
(can be used multiple times)
--vmware-tsc-map    Use VMware TSC map instead of native RDTSC
--proc-type         Type of this process (primary|secondary|auto)
--syslog            Set syslog facility
--log-level=   Set global log level
--log-level=:
Set specific log level
-v                  Display version information on startup
-h, --help          This help
--in-memory   Operate entirely in memory. This will
disable secondary process support
EAL options for DEBUG use only:
--huge-unlink       Unlink hugepage files after init
--no-huge           Use malloc instead of hugetlbfs
--no-pci            Disable PCI
--no-hpet           Disable HPET
--no-shconf         No shared config (mmap'd files)
EAL Linux options:
--socket-mem        Memory to allocate on sockets (comma separated values)
--socket-limit      Limit memory allocation on sockets (comma separated values)
--huge-dir          Directory where hugetlbfs is mounted
--file-prefix       Prefix for hugepage filenames
--base-virtaddr     Base virtual address
--create-uio-dev    Create /dev/uioX (usually done by hotplug)
--vfio-intr         Interrupt mode for VFIO (legacy|msi|msix)
--legacy-mem        Legacy memory mode (no dynamic allocation, contiguous segments)
--single-file-segments Put all hugepage memory in single files
EAL: FATAL: Invalid 'command line' arguments.
29/10/2019 -- 07:44:09 -  - [ERRCODE: SC_ERR_DPDK_CONFIG(319)] -  No DPDK ports found
step in trace from GDB core dump (build with -g): Could you explain more detail how to do this?
Thank you very much
Tri

The hint from your logs is EAL: FATAL: Invalid 'command line' arguments. and not just ERRCODE: SC_ERR_DPDK_CONFIG(319)] - No DPDK ports found.
The application is built with -Wl,-lrte_pmd_vhost  in src/Makeilfe, hence I do not assume this is cause.
You have mentioned in cfg --vdev=virtio_user1,queues=1,path=/var/run/usvhost*

The only place lef tot look is vhost server-side. Can you cross-check and tell me

how are you creating the server instance of vhost?
Is the server instance running, when you run the dpdk-suricata application?
before running the dpdk-suricata, can you ls /var/run/usvhost*
is dpdk applciation like l2fwd, testpmd, or basicfwd runs with --vdev=virtio_user1,queues=1,path=/var/run/usvhost*

If all the 4 is valid, then we can debug what is going wrong

hi @tritm
here is the difference I feel

On the host you run testpmd with dpdk applciation like l2fwd, testpmd, or basicfwd runs with --vdev=virtio_user1,queues=1,path=/var/run/usvhost1 - it works

On the docker instance do you run dpdk applciation like l2fwd, testpmd, or basicfwd runs with --vdev=virtio_user1,queues=1,path=/var/run/usvhost1? If yes does it work? If yes, as part of dockert args you are passing /var/run/usvhost1.

With the above understanding if you are passing both /var/run/usvhost1 and /var/run/usvhost2 for 2 interfaces for docker dpdk-suricata there should not be issue. Can you confirm the same?
Also please share ip address and user login details for ssh

I am waiting for your updates and login details. Please share

@vipinpv85
2. On the docker instance, I run testpmd with --vdev virtio_user1,queues=1,path=/var/run/usvhost1
and it works
3. About my server log in details, do you use any other private channel as: Telegram so I can share it with you?
BR
Tri

I am sure how telegram can do ssh port or socks forward. If you are talking about suricat.log you can try to increase dpdk verbose '--log-level=eal,8' abd analyze wgat is going wrong

@vipinpv85 I would like to send you my ssh details to login to my server via a private channel such as Telegram, or any other that is convinient to you

I got this error when re-install it:
I dont know why it relates to memory
root@414b32b6e041:~/DPDK_SURICATA-4_1_1/suricata-4.1.4# suricata --dpdk=mysuricata.cfg
1/11/2019 -- 07:35:11 -  -  DPDK Mode selected
1/11/2019 -- 07:35:11 -  - This is Suricata version 4.1.4 RELEASE
*** Error in `/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata': malloc(): memory corruption: 0x0000000000fa3ca0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ffff523d7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7ffff524813e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7ffff524a184]
/lib/x86_64-linux-gnu/libc.so.6(__strdup+0x1a)[0x7ffff525148a]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(eal_parse_common_option+0xe78)[0x540728]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(rte_eal_init+0xf6)[0x528956]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(main+0x2287)[0x4ae467]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffff51e6830]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(_start+0x29)[0x4b1539]
======= Memory map: ========
00400000-00bfd000 r-xp 00000000 fc:01 3151871                            /root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata
00dfd000-00e09000 r--p 007fd000 fc:01 3151871                            /root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata

@vipinpv85 I would like to send you my ssh details to login to my server via a private channel such as Telegram, or any other that is convinient to you

You can rachme in linkedin, gmail or skype id is vipinpv85

I got this error when re-install it:
I dont know why it relates to memory
root@414b32b6e041:~/DPDK_SURICATA-4_1_1/suricata-4.1.4# suricata --dpdk=mysuricata.cfg
1/11/2019 -- 07:35:11 - - DPDK Mode selected
1/11/2019 -- 07:35:11 - - This is Suricata version 4.1.4 RELEASE
*** Error in `/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata': malloc(): memory corruption: 0x0000000000fa3ca0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ffff523d7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7ffff524813e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7ffff524a184]
/lib/x86_64-linux-gnu/libc.so.6(__strdup+0x1a)[0x7ffff525148a]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(eal_parse_common_option+0xe78)[0x540728]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(rte_eal_init+0xf6)[0x528956]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(main+0x2287)[0x4ae467]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffff51e6830]
/root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata(_start+0x29)[0x4b1539]
======= Memory map: ========
00400000-00bfd000 r-xp 00000000 fc:01 3151871 /root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata
00dfd000-00e09000 r--p 007fd000 fc:01 3151871 /root/DPDK_SURICATA-4_1_1/suricata-4.1.4/src/.libs/lt-suricata

This is suricata app stating it can not malloc the heap space while running in the environment

Thanks for the debug terminal, on the investigation, it is clear the net_vhost is dropping TX packets from host to VM or container. Hence using HOST to Docker is not working on the given environment.
I have tested similar on my enviroment.
Ubuntu: Ubuntu 18.04.3 LTS
uname -r: 4.15.0-65-generic
DPDK: dpdk-stable-18.11.3
Host:
cmd-1: ./build/l2fwd -l 3-4 --vdev 'eth_vhost0,iface=/tmp/sock0' --file-prefix=host --no-pci -m 1024 -- -p 0x1 -T 1
cmd-2: ./build/l2fwd -l 4,5 --vdev 'virtio_user0,path=/tmp/sock0' --file-prefix=host --no-pci --file-prefix=client --vdev net_tap0,iface=test  -- -p 0x3 -T 1
screenshot:

Dear @vipinpv85
It's very kind of you spending time and effort to troubleshoot the problem for me. I'm very appreciate it.
"it is clear the net_vhost is dropping TX packets from host to VM or container. Hence using HOST to Docker is not working on the given environment"
When I check your 2 cmds in my environment, I see

If it is possible, could you help suggest how could I change my environment (change the dpdk version, or change to the new server)? for it to be able to install suricata dpdk and test it?
Thank you very much
Tri

Hi Vipin,
I am Yasufumi Ogawa. I am trying to run suricata on container with Tri.
Could I confirm how you launch container itself because vhost is working
on my environment.
Here is my results by using l2fwd and testpmd, though I am using DPDK
v18.08 on host and v19.08 for some reason. I can run other than testpmd
on container such as l3fwd or pktgen. So, I think suricata is also
possible to run on container. I would appreciate if you give we some
feedback.
Host:

Ubuntu 18.04
DPDK v18.08
5.0.0-32-generic
cmd-1 (l2fwd on host):
sudo ./build/l2fwd -l 0-2 --vdev 'eth_vhost0,iface=/tmp/sock0'
--file-prefix=host --no-pci -m 1024 --vdev 'eth_vhost1,iface=/tmp/sock1'
-- -p 0x3

Container:

Ubuntu 18.04
DPDK v19.08
5.0.0-32-generic
cmd-2 (testpmd on container):
sudo docker run
-it
--privileged
-v /tmp/sock0:/var/run/usvhost0
-v /tmp/sock1:/var/run/usvhost1
-v /dev/hugepages:/dev/hugepages
sppc/dpdk-ubuntu:latest
testpmd
-l 3-5
-n 4
-m 1024
--proc-type auto
--vdev virtio_user0,queues=1,path=/var/run/usvhost0
--vdev virtio_user1,queues=1,path=/var/run/usvhost1
--file-prefix spp-testpmd-container0
--
--interactive

Regards,
Yasufumi
On 2019/11/04 13:37, Vipin Varghese wrote:

Thanks for the debug terminal, on the investigation, it is clear the
net_vhost is dropping TX packets from host to VM or container. Hence
using HOST to Docker is not working on the given environment.
I have tested similar on my enviroment.
Ubuntu: Ubuntu 18.04.3 LTS
uname -r: 4.15.0-65-generic
DPDK: dpdk-stable-18.11.3
Host:
cmd-1: ./build/l2fwd -l 3-4 --vdev 'eth_vhost0,iface=/tmp/sock0'
--file-prefix=host --no-pci -m 1024 -- -p 0x1 -T 1
cmd-2: ./build/l2fwd -l 4,5 --vdev 'virtio_user0,path=/tmp/sock0'
--file-prefix=host --no-pci --file-prefix=client --vdev
net_tap0,iface=test -- -p 0x3 -T 1
screenshot:
image
https://user-images.githubusercontent.com/1296097/68099857-da502a80-feea-11e9-8afe-eaddb1e3b2d2.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vipinpv85/DPDK_SURICATA-4_1_1/issues/5?email_source=notifications&email_token=AG6DL3XAHTIHCG5BDPME47LQR6RHRA5CNFSM4ITH5Q5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC6HS2A#issuecomment-549222760,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG6DL3QA5DREQLYNOCBKNSDQR6RHRANCNFSM4ITH5Q5A.

Hi Vipin, I am Yasufumi Ogawa. I am trying to run suricata on container with Tri. Could I confirm how you launch container
[VV] during my debug with Tri, we have not launched in a container

itself because vhost is working on my environment.
[VV] that is good. now you can try running CMD-2 in the container with path shared to VM

Here is my results by using l2fwd and testpmd, though I am using DPDK v18.08 on host and v19.08 for some reason.
I can run other than testpmd on container such as l3fwd or pktgen.
[VV] on host you have DPDK v18.08 and container you using v19.0. From host packet are sent to cotnainer. So your ideal
enviroment should be
Host: Physical interface ==> DPDK app like example/skeletion or example/l2fwd to forward to net_vhost0.
Container: vhost_user0 recieving traffic from host.
test scenario can be
Host: TAP interface with PKTGEN ==> DPDK app like example/skeletion or example/l2fwd to forward to net_vhost0.
Container: vhost_user0 recieving traffic from host.
So, I think suricata is also possible to run on container. I would appreciate if you give we some feedback.
[vv] shared above
Host:

Ubuntu 18.04
DPDK v18.08
5.0.0-32-generic
cmd-1 (l2fwd on host): sudo ./build/l2fwd -l 0-2 --vdev 'eth_vhost0,iface=/tmp/sock0' --file-prefix=host --no-pci -m 1024 --vdev 'eth_vhost1,iface=/tmp/sock1' -- -p 0x3
[VV] this looks like the modified command I have shared earlier, but with few differences

a. -l 0-2: you should avoid running on core 0
b. --no-pci: this will prevent any PCIe device from used in the applications.
c. --vdev 'eth_vhost1,iface=/tmp/sock1': I assume this is interface you want to add to pktgen. But I do not find the same shared.
Container:

Ubuntu 18.04
DPDK v19.08
5.0.0-32-generic
cmd-2 (testpmd on container): sudo docker run -it
--privileged
-v /tmp/sock0:/var/run/usvhost0
-v /tmp/sock1:/var/run/usvhost1
-v /dev/hugepages:/dev/hugepages
sppc/dpdk-ubuntu:latest
testpmd
-l 3-5
-n 4
-m 1024
--proc-type auto
--vdev virtio_user0,queues=1,path=/var/run/usvhost0
--vdev virtio_user1,queues=1,path=/var/run/usvhost1
--file-prefix spp-testpmd-container0
--
--interactive
[VV]

I think you are using testpmd because; you have used apt isntall dpdk* hence the app testpmd. You can pass your DPDK folder instead.
you are passing both usvhost1 and usvhot0 from sock1 and sock0. Which means you no means to generate traffic either from host or cotnainer to test.
3, --proc-type=auto: container is seperate instance, hence this should be primary always.

question: why can not you pass the args as -vdev 'virtio_user0,queues=1,path=/var/run/usvhost0' --vdev net_tap0,iface=test? This will create a tap interface in container.
information: I think for testpmd you have to pass portmask and in interactive shell start tx_first

@yasufum you can refer to the link which I had shared yesterday to TRi https://doc.dpdk.org/guides/howto/virtio_user_for_container_networking.html

so please ensure, cmd-1 and cmd02 works on host. If yes, then terminate cmd-2 and run on cotnainer

@vipinpv85
Thank you for your comments.

I think you are using testpmd because; you have used apt isntall dpdk* hence the app testpmd. You can pass your DPDK folder instead.
Sorry for lack of explaining about my env. In my container, I have built DPDK from source and set RTE_SDK.

Which means you no means to generate traffic either from host or cotnainer to test.
Container has two ports, rx and tx, and traffic is generated in testpmd containr. As shown in pictures I sent your gmail, it looks working fine.

why can not you pass the args as -vdev 'virtio_user0,queues=1,path=/var/run/usvhost0' --vdev net_tap0,iface=test?
It is because simply tap is not needed to my test case.

you have to pass portmask and in interactive shell start tx_first
Yes. portmask is dropped from my log. sorry.
I am using testpmd in interactive mode, and traffic is generated with start tx_first.

so please ensure, cmd-1 and cmd02 works on host. If yes, then terminate cmd-2 and run on cotnainer
OK. Thanks

@yasufum your colleague Tri has pinged me stating you have new setup and it is working. Please avoid double information as it is confusing.

updating for future reference, Minutes of troubleshooting

Tried on 16.04 and 18.04 Ubuntu release, both environment test scenario failed.
Ran with both dpdk-stable-17.11.7 and dpdk-stable-18.11.2 (failed).
Found problem a. for new console default path is dpdk and not 'dpdk-stable-*' (corrected the same)
Found problem b. running from build folder is not able to fetch address. (corrected by moving to the example parent folder).
increased the logging to '--log-level=pmd,8' found @tritm enviroment is not able to connect via eventfd.

For working, the logs should be
VHOST_CONFIG: new vhost user connection is 17
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:18
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:19
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region 0, size: 0x40000000
         guest physical addr: 0x7f0040000000
         guest virtual  addr: 0x7f0040000000
         host  virtual  addr: 0x7f5b40000000
         mmap addr : 0x7f5b40000000
         mmap size : 0x40000000
         mmap align: 0x40000000
         mmap off  : 0x0
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:21
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:22
VHOST_CONFIG: virtio is now ready for processing.
PMD: New connection established
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
PMD: vring0 is enabled
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
PMD: vring1 is enabled

@vipinpv85 I changed to other server but still got the same problem

dpdk-stable-18.11.3

@tritm did you check what I asked you to check?
@yasufum have you cross-checked did cmd1 and cmd2 really worked on your setup?

the reason for your error is vhost driver is not getting vring_state_changed

@tritm and @yasufum, while exploring more onto your problem I came across https://www.dpdk.org/wp-content/uploads/sites/35/2018/12/YasufumiOgwa_Implementing-DPDK-based-Application-Container-Framework-with-SPP.pdf. This helps me understand why you were using ubuntu 16.04 and dpdk 18.02.
there is an interesting point shared Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8) hugepages. If you have more regions (especially when 2MB hugepages are used), the option, –single-file-segments, can help to reduce the number of shared files. Since your enviroment has around 5K 2MB huge pages, you can start by allocating 512MB in which cmd1 uses '-m 256' and cmd2 uses '-m 256'

@vipinpv85 have set the vt-d enabled on the server and already set intel_iommu=on but the results are still the same

@vipinpv85 have set the vt-d enabled on the server and already set intel_iommu=on but the results are still the same

thanks for confirming, did you try what is been updated `@tritm and @yasufum, while exploring more onto your problem I came across https://www.dpdk.org/wp-content/uploads/sites/35/2018/12/YasufumiOgwa_Implementing-DPDK-based-Application-Container-Framework-with-SPP.pdf. This helps me understand why you were using ubuntu 16.04 and dpdk 18.02.
there is an interesting point shared Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8) hugepages. If you have more regions (especially when 2MB hugepages are used), the option, –single-file-segments, can help to reduce the number of shared files.
Or since your enviroment has around 5K 2MB huge pages, you can start by allocating setting 512 2MB instead of 5k2MB. In which cmd1 uses '-m 256' and cmd2 uses '-m 256'`

@yasufum are you also having same host problem?

updating for future reference and others:
had skype debug with @tritm, requested multiple times to use the option from  https://www.dpdk.org/wp-content/uploads/sites/35/2018/12/YasufumiOgwa_Implementing-DPDK-based-Application-Container-Framework-with-SPP.pdf. Due to some personal @tritm does not want to try to fix the 4th issue in the environment.
Hence I am unable to debug any further.  Test Vhost interface with DPDK examples on ubuntu 16.04 and 18.04 with DPDK 18.11.3 and higher in VM, container and Host. All combination works.

@vipinpv85 The vhost work now with the following 2 commands:
cmd-1: ./l2fwd -l 3-4 --vdev 'eth_vhost0,iface=/tmp/sock0' --file-prefix=host --no-pci -m 256 --single-file-segments -- -p 0x1 -T 1
cmd-2: ./l2fwd -l 2,4 -m 256 --vdev 'virtio_user0,path=/tmp/sock0' --no-pci --file-prefix=client --single-file-segments --vdev net_tap0,iface=test -- -p 0x3 -T 1

Thank you very much for you effort to help
So now how can I do with suricata?
Tri

@vipinpv85 The vhost work now with the following 2 commands:
cmd-1: ./l2fwd -l 3-4 --vdev 'eth_vhost0,iface=/tmp/sock0' --file-prefix=host --no-pci -m 256 --single-file-segments -- -p 0x1 -T 1
cmd-2: ./l2fwd -l 2,4 -m 256 --vdev 'virtio_user0,path=/tmp/sock0' --no-pci --file-prefix=client --single-file-segments --vdev net_tap0,iface=test -- -p 0x3 -T 1

Thank you very much for you effort to help
So now how can I do with suricata?
Tri

Execute command 2 in docker while allowing command 1 on host

@tritm and @yasufum can you share your updates. If no support is required, please do say so.

@tritm and @yasufum; since there are no updates from your end I assume you got your environment working. but neither @tritm or @yasufum has updated the ticket so far. Not surprising based on the earlier interaction. Let the content be out there which might help people to debug.

@vipinpv85 , sorry for making no response for long time. Although I might not catch up everything after I dropped out, I understand that @tritm cannot run l l2fwd because for the restriction of 2M  hugepages.
On my environment on which I have 1G hugepages, I can run both of l2fwd on host and container, although some options are required for dokcer.
After run "cmd-1" on host and launch a container as following, I succeeded to run "cmd-2" as you suggested. I used DPDK v18.08.
docker run --privileged -v /tmp/sock0:/tmp/sock0  -v /dev/hugepages:/dev/hugepages -it sppc/dpdk-ubuntu /bin/bash

I would like to ask @tritm that he still need your support. Thank you a lot for your kindness.

@yasufum thank you for the update. Please find the comments below
I understand that @tritm cannot run l l2fwd because for the restriction of 2M hugepages
[VV] @tritm is able to run on host machines. Have suggested to run cmd2 on container no updates yet. With respect to 2M huge pages, the file-segment option is required for pages more than 512.

@vipinpv85 You are right. I've forgotten the option.

Dear @vipinpv85, Thank you very much for your help.


Ruby compiles from Umple templates rather than JET by default

Changed the build file to use Umple templates by default for ruby. As well, was able to remove all of the generated code for umple ruby templates, since they will always be built.
Like the switch for php, it defaults to using the stable version of umple since it should always be available.

This looks good. Ignore my overnight comment issue #708. I will merge.


Default nginx access log format parse fix

Default nginx access log format (http://nginx.org/en/docs/http/ngx_http_log_module.html) looks like:
log_format compression '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $bytes_sent '
                       '"$http_referer" "$http_user_agent" "$gzip_ratio"';

Fluentd parser was missing "$gzip_ratio".
How to reproduce the bug:
➜  ~ docker run --name test-nginx -P -d nginx
[…]
➜  ~ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                           NAMES
3b445987ac4d        nginx               "nginx -g 'daemon off"   16 hours ago        Up 2 minutes        <IP_ADDRESS>:32769->80/tcp, <IP_ADDRESS>:32768->443/tcp   test-nginx
➜  ~ curl -so /dev/null http://localhost:32769
➜  ~ docker logs 3b445987ac4d
<IP_ADDRESS> - - [01/Nov/2016:01:25:20 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.49.1" "-"

Fluentd was unable to parse default nginx log format without this patch.
My patch adds missing field and keep backward compatibility.
The issue was reported also here - https://github.com/fluent/fluentd/issues/608

@ljagiello It's just example, not default. See "Directives":
Syntax: access_log path [format [buffer=size] [gzip[=level]] [flush=time] [if=condition]];
Default: access_log logs/access.log combined;

Default combined is also described in same page:
The configuration always includes the predefined “combined” format:
log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

@tagomoris @repeatedly
Inside official nginx package - http://nginx.org/packages/mainline/debian/pool/nginx/n/nginx/nginx_1.11.5-1~jessie_amd64.deb
configuration looks now:
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

I wrongly assume it's a gzip ratio, in fact it's $http_x_forwarded_for. IMHO if it's part of official package it should be supported. As a result of that docker/kubernetes containers will be supported by default.


v0.4.0 release for windows is not recognized as executable

Prerequisites
Put an x into the box that applies:

[x] This issue describes a bug.
[ ] This issue describes a feature request.

For more information, see the Contributing Guidelines.
Description
The windows 0.4.0 release is not recognized as an executable.
Actual Behavior
I downloaded the latest windows release and am unable to run the binary on my windows machine.
Expected Behavior
I want to be able to install terraform-docs on my windows machine.
Replication Case
Tell us which steps to take to replicate your problem.

Download 0.4.0 windows release to a windows machine
Execution of the download is prohibited

Hi @willegetz. Do you have any error message to share with us? Have you tried running the executable from v0.3.0? Does that one work?

Here's what the Unix file command has to say:
❯ file ./bin/terraform-docs-v0.4.0-windows-amd64
./bin/terraform-docs-v0.4.0-windows-amd64: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows

Hi @metmajer, thank you for responding.
After downloading the file and attempting to execute it I see the "How do you want to open this file" promt:

The same prompt is launched when I try and execute the file from PowerShell:
Start-Process .\terraform-docs-v0.4.0-windows-amd64
Interestingly enough, when compiling this response, I discovered that when I execute the file from the cmd.exe, I am prompted with the command line parameters:
> terraform-docs-v0.4.0-windows-amd64
    Usage: terraform-docs [--no-required] [--no-sort] [--with-aggregate-type-defaults] [json | markdown | md] <path>...
    terraform-docs -h | --help
Using cmd.exe works to generate terraform docs as described.
Thank you for your time,
William Getz

@willegetz likewise, thanks for the update! Can you please try and see if the executable can be started if you append the .exe extension to the filename?

@metmajer, yes, I can confirm that the Windows release binary is missing its .exe extension and adding this allows the binary to run.

Much appreciated, @X-Guardian!

I apologize for the delay in response @metmajer, and am grateful for @X-Guardian verifying your question.
Thank you both.


Scipy update

We're giving a very short lightning talk on Project Jupyter updates for the Scipy community at scipy this week (3 minutes for all of Jupyter!). If you had 10-20 seconds to communicate one major new thing to the Scipy community, what would it be?

Closing this issue, since SciPy has passed. @jasongrout, @blink1073, others and I worked together to give the Jupyter Server update. Big thank you to @marthacryan


Hs/docker

Fixes https://github.com/mass-matrix/AMPL/issues/9

I apologize. I meant to merge this with my own fork, and opened this PR mistakenly


How to make more better structural branded types

It can create lots of new types easily and structurally.
function build<N, T>(f: (a: T) => T = (a: T) => a) {
	return <{
		(a: N & T): void;
		(a: T): N & T;
	}>f;
}
namespace Types {
  export namespace Nominal1 {
    class Nominal<T> {
      private NOMINAL: T;
    }
    export type A = Nominal<'A'> & number;
	export const A = build<Nominal<'A'>, number>();
    export type B = Nominal<'B'> & number;
	export const B = build<Nominal<'B'>, number>();
  }
}

namespace Types {
  export namespace Nominal2 {
    class Nominal<T> {
      private NOMINAL: T;
    }
    export type A = Nominal<'A'> & number;
	export const A = build<Nominal<'A'>, number>();
    export type B = Nominal<'B'> & number;
	export const B = build<Nominal<'B'>, number>();
  }
}

var a1: Types.Nominal1.A;
var b1: Types.Nominal1.B;
var a2: Types.Nominal2.A;
a1 = b1; // error
a1 = a2; // error
a1 = Types.Nominal1.A(0); // ok
a1 = Types.Nominal1.A(a1); // error
a1 = Types.Nominal1.B(0); // error
a1 = Types.Nominal2.A(0); // error

looks like the same topic as in https://github.com/Microsoft/TypeScript/issues/4895. i suggest moving the discussion there.

This issue posted only for documentation. This is a different technique, and has a different purpose from #4895.

enhanced.

This issue posted only for documentation. This is a different technique, and has a different purpose from #4895.

I understand; #4895 is a discussion item that is meant for brainstorming and sharing ideas.


OnDataPointListener Stops getting updates randomly?

 Issue by RyanNewsom
Friday Jul 22, 2016 at 22:52 GMT
Originally opened as https://github.com/googlesamples/android-fit/issues/17

All,
My team noticed this problem in our application which uses your Sensors API to show users their distance traveled. Every time we start a walk, eventually, the OnDataPointListener just stops giving updates that there's a new DataPoint available(even though user is moving). I've seen it take anywhere from 5 mins to 1.5 hours to stop working. We have log statements tracking when theres an update, when the client gets disconnected(if it ever does), when a client gets built, etc. There is no callback when this happens. I forked your sample app to try and see if this was an API issue, and it looks like it is.
https://github.com/RyanNewsom/android-fit/tree/master/BasicSensorsApi
Steps to reproduce:

Download my sample app, launch, press start.
Confirm updates are coming in
Background the app, hit the hold button to turn off the screen
Put phone in pocket/on a table for 10-30 mins
Foreground app and walk around of shake the phone to see if you are getting updates
repeat 3-5 until you see you are not getting updates

🙏

 Comment by RyanNewsom
Tuesday Jul 26, 2016 at 21:30 GMT

Today I did some testing and I see a lot of this in logcat...... I'm sure this is whats causing the issue.
07-26 15:11:18.199 5057-30645/? I/Fitness: Found dead listener BinderListener{app.package.name d:distance.delta:gms:SM-G920V:180badfa:live_distance_from_steps android.os.BinderProxy@fa2a5e2}, removing.


Cursor effects not working in v4.5.1

The cursor of sprite not working when I set it.
sprite.cursor = 'text';
https://jsfiddle.net/sd1ebmjg/1/

In version 4.3.0, we added the cursorStyles dictionary to the InteractionManager, to allow even more flexibility in cursor usage. The cursor property on your sprite is used to select which what to use from cursorStyles.
Add the following to your jsfiddle to make it work correctly:
app.renderer.plugins.interaction.cursorStyles.text = 'text';

However, we should make a patch to use the cursor mode as a CSS value if there is nothing present in cursorStyles.

Thank you very much!


Fix pallet_offences_benchmarking::check_events

pallet-offences-benchmarking has an internal function check_events that is currently only enabled for tests. When you try to use this function with a real chain like Kusama it fails. It would be nice to have this fixed and working for all kind of chains. This ensures that the benchmarking is done correctly.

I would like to work on this.

You are free to work on this @gitofdeepanshu!

This is exactly what this issue is about. If I would have known the issue, I would have fixed it right away :D


Don't skip Javadoc when building agent

Currently, the Javadoc task gets skipped when building the agent. If there's an error in the Javadoc we only learn that during the release phase which is not ideal.
I'm not sure if there was a reason behind skipping the Javadoc or if it was a leftover from setting up the pipeline.

Actually there is a dedicated stage in the pipeline for javadoc generation, the part you changed just avoids generating javadoc in the build (see line 253 of the pipeline).

Good call, updated to enable the Javadoc pipeline step instead

Codecov Report

Merging #1137 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1137      +/-   ##
============================================
- Coverage     59.64%   59.61%   -0.03%
  Complexity       85       85
============================================
  Files           328      328
  Lines         14891    14891
  Branches       2086     2086
============================================
- Hits           8881     8878       -3
- Misses         5407     5412       +5
+ Partials        603      601       -2

Impacted Files
Coverage Δ
Complexity Δ

...pm/agent/profiler/asyncprofiler/AsyncProfiler.java
60.46% <0.00%> (-4.66%)
0.00% <0.00%> (ø%)

...o/elastic/apm/agent/profiler/SamplingProfiler.java
77.65% <0.00%> (-0.28%)
0.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a1587e...96c6ae5. Read the comment docs.


talk proposals: add some configurable instructions

This is useful to specify accomodation and transportation terms for instance.

More specifically, when configuring a conference, it should be possible to add severals questions:

for the speaker (exemple: city of origin)
for each talk (exemple: sound required)
Ideally, it should be possible to configure for string or boolean answer.


Trigger tests when a PR is submitted

Currently only looks for PRs to the main branch but can be added for other branches. Also ignores updates to docs/README. The test results should show up in the PR discussion as well

It's strange because it worked 5 minutes before on the PR to my fork. I submitted the PR to this repo and the tests automatically ran (which I don't think they should have based on the existing pipeline?). It seems like coveralls is having issues (https://github.com/lemurheavy/coveralls-public/issues/1710) but all the examples I can find are with github actions. There is this potential workaround but not sure if it will work. Do you know if there are already existing pipelines associated with the repo?

Looking at previous PRs it doesn't seem like any test suites ran automatically. I wonder if coveralls was complaining because it was running against the same commit message?

I admit to not really knowing the issues here; I kind of cargo-culted together stuff like coveralls, and have very little understanding of it all. I don't recall coveralls complaining about the same commit message before though.

Let's just merge for now and sort out coveralls later ...


NullReferenceException when calling StartMonitoring

PreparedSqlLiteInsertCommand.ExecuteNonQuery (System.Object[] source)
System.NullReferenceException: Object reference not set to an instance of an object
Please check all of the platforms you are having the issue on (if platform is not listed, it is not supported)

[] iOS
[X] Android
[] UWP
[] .NET Standard

Version of OS(s) listed above with issue
Android 8.0.0
Android 7.0
Version of Library
3.0.0-beta3
Expected Behaviour
No Crash
Actual Behavior
If applicable, please include entire exception - type, message, stacktrace
PreparedSqlLiteInsertCommand.ExecuteNonQuery (System.Object[] source)
SQLiteConnection.Insert (System.Object obj, System.String extra, System.Type objType)
SQLiteConnection.Insert (System.Object obj)
GeofenceManagerImpl+d__10.MoveNext ()
AsyncMethodBuilderCore+<>c.b__6_1 (System.Object state)
QueueUserWorkItemCallback.WaitCallback_Context (System.Object state)
ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx)
ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx)
IThreadPoolWorkItem.ExecuteWorkItem ()
ThreadPoolWorkQueue.Dispatch ()
_ThreadPoolWaitCallback.PerformWaitCallback ()
Code of Conduct

No Library Update Requests
No Timeline Requests
Please read through the documentation and FAQs
Please be respectful of my time. I put a lot of effort into my libraries.  I try to support the issue requests within a day.  I'm not psychic, so please take the time to fill out the above form as complete as possible

If you are testing betas, can you please include an example of the call(s) you made?  Did you request permissions?  Did you init the library, etc?

Here is the function, which is called in Application Startup
    /// <summary>
    /// Get Current Location based on Geo Location and Start Monitoring for each fence (location)
    /// </summary>
    /// <param name="geofences">Instance of GeoFences if null, function will not start monitoring for each location</param>
    /// <param name="currentPosition">Current Position if null, function will get it</param>
    /// <returns>
    /// True if Geo Location Found; otherwise false
    /// </returns>
    internal async Task<bool> GetCurrentGeoLocationAsync(
        IGeofenceManager geofences = null, Position currentPosition = null)
    {
        bool hasGeoLocation = false;
        if (currentPosition == null) currentPosition = await GetCurrentGeoLocation();
        if (EntityList != null && currentPosition != null)
        {
            var hasRegions = false;
            if(geofences != null)
            {
                geofences.StopAllMonitoring();
            }
            foreach (var location in EntityList)
            {
                try
                {
                    if (location.Latitude > -90 && location.Latitude < 90 &&
                        location.Longitude > -180 && location.Longitude < 180 &&
                        location.TotalMeters > 0 && location.Longitude < 3000)
                    {
                        var radius = Distance.FromMeters(location.TotalMeters);
                        var center = new Position(location.Latitude, location.Longitude);
                        var geoRegion = new GeofenceRegion(location.Name, center, radius);
                        // If Current Location is inside one of the Geo Fence, Update Device Location
                        if (currentPosition != null && geoRegion.IsPositionInside(currentPosition))
                        {
                            if(SetDeviceLocationFromGeoLocation(geoRegion))
                            {
                                hasGeoLocation = true;
                            }
                        }
                        // Start Monitoring for this Geo fence if geo fence instance is given
                        if (geofences != null)
                        {
                            try
                            {
                                geofences.StartMonitoring(geoRegion);
                                hasRegions = true;
                            }
                            catch { }
                        }
                    }
                }
                catch (Exception ex)
                {
                    this.Log().Error($"Exception while setting geo fence {location.Name}", ex);
                }
            }
            // Register to listen for changes to geo fences
            if (hasRegions && geofences != null)
            {
                geofences.RegionStatusChanged += Geofences_RegionStatusChanged;
            }
        }
        return hasGeoLocation;
    }

where geofences is CrossGeofences.Current;
and currentLocation is from await GetCurrentGeoLocation();
    /// <summary>
    /// Get Current Geo Location
    /// </summary>
    /// <returns>Position</returns>
    private async Task<Position> GetCurrentGeoLocation()
    {
        try
        {
            var hasPermission = await PermissionChecker.CheckPermissions(Permission.Location);
            if (!hasPermission)
                return null;

            var request = new GeolocationRequest(GeolocationAccuracy.Medium, TimeSpan.FromMilliseconds(10000));
            var position = await Geolocation.GetLocationAsync(request);
            if (position != null)
                return new Position(position.Latitude, position.Longitude);
        }
        catch (Exception ex)
        {
            LogError($"Cannot get GPS Location of Device {ex.Message}", ex);
        }
        return null;
    }

I have following permission requested in AndroidManifest

@aritchie Let me know if you need additional information


[Site] Style prop docs

The prop doc table is ugly. Make it look nicer.
How do we best document default values, required values, prop type, and prop description.


[typescript] Change Transition.children to type ReactNode

Children should be typed as a ReactNode not a ReactElement.

I don't think so https://github.com/DaIgeb/material-ui/blob/e8239fc9339ef345bc6f0b18e689ec80f96b89a3/src/internal/Transition.js#L401.


LSP Request Forwarding: completion item documentation is lost during resolve

I am trying to add request forwarding for my language server which is a completion item provider with resolve support enabled. The request forwarding for completion works as shown in the sample extension when resolve support is disabled, but since I must have resolve support for my language server, the documentation for the completion items generated via request forwarding gets lost during the resolve step. Since there's no way to be a "partial" resolve provider or to return undefined in the LSP resolve provider to signal no resolve was required (which is possible in VS Code), I'm not sure how to avoid this loss.
Steps to reproduce:

Download and install this modified version of the lsp-embedded-request-forwarding sample extension. The sample was modified to enable completion item resolve support (the completionItem/resolve handler just returns the item it receives) and to use the latest vscode-languageserver/client modules (8.0.2) and vscode-html-languageservice (5.0.1).
lsp-embedded-language-service-sample-1.0.0.vsix.zip
Create a file called test.html1 with the following content:

This is a heading

This is a paragraph.

With the modified sample extension enabled, jump to line 4 and column 22 in the file and trigger completion using Ctrl-Space.
Observe that the documentation for background-color is lost:

Continuation of #151537

This can be fixed by adding a handler for the resolveCompletionItem property in the language client middleware that immediately returns the CompletionItem if it contains documentation.


Investigate Path Details memory usage and performance

Currently there is a measurable slow down of nearly 10% when instructions are enabled. (In the Measurement class we also enable the PathDetails calculation if withInstructions=true).
So we should look if everything is good when looking into the memory and speed via a profiler. Also see if the slow down disappears if path details are disabled.

I'm pretty sure that there is no slow down if we disable path details (we still have to check), but I would like to be sure that when we enable them the overhead is as low as possible.

Ah OK, so this is only when we enable PathDetails? 10% for CH sounds actually not too unreasonable to me, when using a couple of PathDetails. The simplification becomes more complicated, which I guess should be a big part of the slow down. The more PathDetails are requested, the more complicated the simplification becomes. I think I would have expected 5% slow down, just as a gut feeling. For example the edgeId PathDetails, should add a lot more simplification runs.

Hmmh, even if I disable path details the same slow down happens, this should not be. Can you confirm this?

I compared the current master to a state before the PathDetails and indeed the routingCH.mean is about 10% higher in the current master.
I did some profiling and found that the method PointList#copy takes too much CPU time. The method is called frequently in PathSimplification#reduceLength. This call is removed in #1138, here.
Also the method DouglasPeucker#subSimplify and DouglasPeucker#compressNew take up a lot of cpu time. When comparing it to the older state of GraphHopper, the simplify algorithm took a lot of cpu time before, so this is probably OK.
So I think we could improve the performance having a more efficient PointList#copy method. I will try this.

We can close this via #1171


Fixes for C and C++ functions and namespaces

This fixes #1561 and also fixes #1719.
Fix lexing of function names (#1561) by adding a keywords state for matching keywords, inside and outside functions.
Before this, when a keyword would appear the lexer would go to the statements state, in which functions were not matched, so functions prefixed with keywords were not matched.

Apart from the c and cpp lexers, I only updated the swig lexer and made sure it's keywords are in the new keywords state, as it was the only lexer that was breaking from this change.

The new keywords state is (almost) entirely optional on purpose for deriving lexers. They don't benefit from using it themselves. Because of that I think it's a waste of time to incorporate this state in deriving lexers so I decided against it.

Fix matching functions with a name or return argument that includes a '::' (also #1561) by introducing a second identifiers regex that matches all the previous identifiers, and also '::', and use it to match for function names and return arguments.

I took the decision to create a second identifiers regex with '::' inside, simply because using the old identifiers regex would hurt
performance massively on every solution I tried to craft. I think this solution is still pretty elegant though, and the issue it fixes is big.

Fix lexing of namespaces (#1719) wherever possible, by adding a namespace state that is entered each time the namespace keyword is matched, and lexes all names matched as namespaces.
Cases this approach doesn't cover:

Namespaces in using declarations.
Namespaces that prefix names in random code.

Unfortunately, in both of these cases the names before and after '::' are not always namespaces so these cases can't be covered.

Merged, thanks a lot for your contribution! Nicely done!


rapidly repeated DNS query

Describe the bug
Ran the below query for cloudwatch log streams last week.  After a few moments, received an error stating my DNS server is "misbehaving."  I'm not clear if that error text is from Steampipe, an AWS API response, or something else.
As it turns out, my own local DNS server was the source of the timeout, there were 1000 DNS queries made from my laptop to the DNS server in less than 60 seconds.    For a few minutes, my laptop is completely cut-off from being able to query DNS, at all, and I lose general connectivity.  I'm basically banned from using DNS for hammering it a thousand consecutive times with the same query.
Steampipe version (steampipe -v)
steampipe (mac) v0.15.0
aws plugin v0.66.0
To reproduce
steampipe query 'select region,first_event_timestamp,last_event_timestamp,log_group_name,name from my_connection_name.aws_cloudwatch_log_stream' --output csv > log-streams.csv
Expected behavior
a csv file from "my_connection_name.aws_cloudwatch_log_stream" containing the fields from my select statement
Additional context

I was able to get around this by adding logs.us-east-1.amazonaws.com to my hosts file so no queries go to my DNS server, but I don't feel like this behavior of rapidly repeating the same DNS query is efficient.
querying a smaller AWS account with less streams seems to be successful.

Including a screenshot of my console output as well as my local eth & wifi ip addresses getting temporarily banned on my home pi-hole DNS server.

Hey @jhodal , in the connection you're testing, which regions have you included, is it just us-east-1? Also, approximately how many log streams do you anticipate you have across all of those regions?
In terms of number of requests, in the aws_cloudwatch_log_stream table, we request up to 50 log streams per request (https://github.com/turbot/steampipe-plugin-aws/blob/main/aws/table_aws_cloudwatch_log_stream.go#L118-L120), which is the maximum allowed by the AWS API. So in terms number of DNS requests, I wonder if we're potentially looking at 50k+ number of log streams?

Hi @cbruno10 thanks for investigating with me.  According to my aws.spc file, the following regions would be queried:

us-east-1
us-east-2
us-west-2

I would suspect 99% of the streams would be found in us-east-1.
As for your other question, i don't know how many log streams there would potentially be; ironically that's one of the questions I'm trying to answer before we start cleaning this up.  But to indirectly answer your question, I have an incomplete CSV output (probably used my modified hosts file to get farther) and I'm seeing 84k lines in that file.  So it looks more than likely I'm hitting limits on the AWS API side once I circumvent the DNS bottleneck.  Well over the 50k+ records you were hinting at.
For what it's worth, i'm not overly concerned about getting the complete list at this point.  In a roundabout way, the answer to how many streams we have turns out to be "way too many streams."  But I am curious about the nature of the rapid/repeat DNS query.
Anyway, please let me know if there is any other info I can provide.  Also, this is a reproducible behavior.

Thanks @jhodal for the information! As a workaround for now, are there any patterns to your log groups that you can use to break up some of the queries? For instance, instead of running something like select * from aws_cloudwatch_log_group, could you run the following queries sequentially:

select * from aws_cloudwatch_log_group where name_prefix = '/aws/lambda/';
select * from aws_cloudwatch_log_group where name_prefix = '/aws/macie/';
select * from aws_cloudwatch_log_group where name_prefix = 'OtherPrefix';

This may help to slow down the outgoing DNS requests and prevent the server from flagging potential abuse.
Please note that in the currently available AWS plugin version, there's a bug in the aws_cloudwatch_log_group table that prevents querying on the name prefix correctly, so I've staged a fix in https://github.com/turbot/steampipe-plugin-aws/tree/update-cw-log-group-prefix-qual. If you'd like to try this branch out, please clone the repo and run make inside of the cloned repo to locally build a plugin version.

@kaidaguerre Based on the data provided by @jhodal , the number of requests seems to be expected (84k/50 = 1680+ requests). In the AWS Go SDK (v1 and v2), I couldn't find any way to limit the number of connections or slow down how many requests are allowed in a given timeframe.
Previously we had thrown around ideas of limiting how many connections Steampipe can create at once, though I'm not sure that would help here since this table is not doing any hydrate calls, but instead just calling the DescribeLogGroups API page after page.
Is there anything available in the Steampipe SDK that I'm missing that could help in this scenario? Or is this something we should instead handle in the AWS plugin code specifically?

@cbruno10 What has me curious is why this creates so many DNS requests specifically? I understand why it would be a lot of AWS requests, but I don't understand why the same DNS name would need to be resolved 1000+ times.
That DNS resolution would be from the AWS plugin down I think - any ideas or search results on the DNS resolution in particular?

@e-gineer Good question, I checked the table code and the code that creates the CloudWatch Logs service client as a starting point, both look as expected and I verified the service client is getting cached correctly. Initial searches across the AWS SDK v1 and v2 issues haven't turned up anything yet about potential DNS resolution issues, but I'll keep searching and see if I can find anything in the issues or code.

@cbruno10 , I re-ran the query as per your suggestion, with a filter (for %lambda%) the query gets interrupted from aws .  The outputted csv file is 1 byte.  So it's unclear if the filter makes a difference.  it simply looks like AWS times-out a query taking too long.
Just for the heck of it, I ran the original query to get the originally reported DNS error (I'm trying to capture logs from the DNS server to get more insight) and my "1000 queries" error is not occurring.  The output is Error: ThrottlingException: Rate exceeded  which is probably from AWS.  113mb.  <yikes!>

I feel there's a variable I'm not aware of that was happening Friday but not today.  I did remove my entry from the hosts file and flushed my DNS before running the steampipe queries in the below screenshot so I'm surprised the originally reported behavior is not happening anymore.
With that said, it's more likely that the source of the original issue IS on my side, not something related to steampipe... i just don't know what it can be at the moment.  Unless I can reproduce it again (and present better, more detailed logs) I don't want to spend your time on something that is plausibly a false alarm.
I would say let's close this issue for now.  If i can reproduce it again we can examine it but the last thing I want is involve you folks to chase a ghost.  Sorry for the runaround.
Did want to take a moment to add Steampipe has become an everyday tool for me.  I appreciate what you guys are putting out there!

@jhodal If you're running into throttling exceptions, what you may find helpful is tuning the config args around error retries, in particular max_error_retry_attempts and min_error_retry_delay. So for instance, you can try increase both of those so we'll try more before giving up when encountering retryable errors (like throttling), and we'll start with a higher initial backoff period. For more information, please see AWS Plugin Configuration.
I'll close this issue for now, but if you're still running into DNS or heavy throttling issue, feel free to re-open or open a new issue and we'll be happy to take a look.
Also, glad to see Steampipe has proven useful for you and thanks for the kind words!


Assertion `ClassTemplate->ProtoLengths[ActualProtoNum] < MAX_PROTO_INDEX' failed

My setup:
tesseract 4.1.0
 leptonica-1.74.4
  libjpeg 8d (libjpeg-turbo 1.4.2) : libpng 1.2.54 : libtiff 4.0.6 : zlib 1.2.8

I have tried with command line tesseract and getting the error. command used:
tesseract img.png out --oem 0 -l eng
Error trace:
Tesseract Open Source OCR Engine v4.0.1 with Leptonica Warning: Invalid resolution 0 dpi. Using 70 instead. Estimating resolution as 389  tesseract: intmatcher.cpp:1160: void ScratchEvidence::UpdateSumOfProtoEvidences(INT_CLASS, BIT_VECTOR): Assertion ClassTemplate->ProtoLengths[ActualProtoNum] < MAX_PROTO_INDEX' failed. Aborted (core dumped)
I have used Tessdata files given on this link: https://github.com/tesseract-ocr/tessdata
As well as, given here: https://github.com/tesseract-ocr/tesseract/wiki/Data-Files , both 4.00 and 3.04/3.05

Using different traineddata files, same error occurs on different images. In both cases, it works perfectly on most of the images, but produces the above error on some images.

There is nothing like Tesseract Open Source OCR Engine v4.0.1. For issue posting use official releases or master code  and tesseract executable not wrapper (we do not support 3rd party projects).

There is nothing like Tesseract Open Source OCR Engine v4.0.1. For issue posting use official releases or master code and tesseract executable not wrapper (we do not support 3rd party projects).

@zdenop  I have updated my tesseract and now I am using official release 4.1.0. I am still getting the same error.

Please provide all information which is needed to reproduce your problem. Without that, nobody can help you. Windows? Linux?`Self-built executable or package from distribution? Image?

Possibly related: issue #444.

Possibly related: issue #444. @xploiter-projects, one or two of the images which trigger the problem would be really helpful. You can attach them here.

@stweil I have attached the sample image. and also updated my system info in the issue

I tested that sample image with Debian package tesseract-ocr 4.1.0-2 and with Tesseract 5.0.0-alpha-337-gc757b (both using tessdata/eng.traineddata) and had no problem.

Please check that you use the latest version of tessdata/eng.traineddata and that the file is not corrupted.  With >4.0 you can try also best and fast repository.
BTW: it seems like your system has not support for AVX, SSE instruction.  Can you provide details how did you build tesseract?

@stweil I actually uploaded cropped image since I didn't want to expose the header.
Anyways, I tested on that cropped image too and it worked fine.
So, this is shape of my image (which is causing error):

and I cropped the image (removed first 300 rows), new shape:

Tesseract is working fine with the cropped image. The area which was cropped didn't contain any text. And it doesn't matter if I crop from bottom or top, it works fine.
But
tesseract works fine with most of the images without cropping/ reducing dimensions. Provided that all of my test images are of same shape (width = 2550, height = 3301)

Please check that you use the latest version of tessdata/eng.traineddata and that the file is not corrupted. With >4.0 you can try also best and fast repository.
BTW: it seems like your system has not support for AVX, SSE instruction. Can you provide details how did you build tesseract?

@zdenop I cloned this branch: https://github.com/tesseract-ocr/tesseract/tree/4.1
and used following commands to build the tesseract:
./autogen.sh
./configure --enable-debug
make
sudo make install
sudo ldconfig

@xploiter-projects, I am confused. Which image can I use to reproduce the problem?

With >4.0 you can try also best and fast repository.

Those won't work because of --oem 0.

@stweil : I know, but I would like to know if problem is in lang model or somewhere else.
@xploiter-projects : if problem is in image, maybe there is problem with memory. What is your hardware specification?

@xploiter-projects : --enable-debug is useful only for developers and it "enable" asserts when specified conditions are met. Unless you understand tesseract code -  do not use this option for build (and you will not see this crash).

@xploiter-projects, if you cannot upload the uncropped image, maybe you can send it per e-mail to me? As @zdenop said, this assertion is not enabled in release code. It is nevertheless an indicator that something possibly goes wrong. Therefore it would be good to examine that case.

@zdenop  @stweil
I built with same method on another system and it seems working fine. I am not sure yet what was the problem. For now, I am closing the issue and I apologize for any inconvenience.  If I encounter this problem again, I will get  back here. Thanks a lot

Please check that you use the latest version of tessdata/eng.traineddata and that the file is not corrupted. With >4.0 you can try also best and fast repository.
BTW: it seems like your system has not support for AVX, SSE instruction. Can you provide details how did you build tesseract?

@zdenop I cloned this branch: https://github.com/tesseract-ocr/tesseract/tree/4.1
and used following commands to build the tesseract:
./autogen.sh
./configure --enable-debug
make
sudo make install
sudo ldconfig


Git Repo Extractor: A new plugin that directly collect data from git repos

Why another git collector?
At the moment, DevLake supports collecting git data through Gitlab/Github API. This approach has two major limitations:

Data coverage issue: Gitlab/Github API only covers a subset of git data
Efficiency / API rate limit issue: for very large repos or an organization with many repos, collecting data through Gitlab/Github API is not the most efficient solution

In this issue, we propose a new plugin that can directly collect data from git repos, which would overcome the limitations described above.
What data to collect?
This new plugin will collect the following entities in the domain layer schema.

repos (repo's metadata)
commits (commit's metadata)
repo_commits (association between repos and commits)
repo_languages (a repo's language composition)
commit_files (file changes associated with a commit)
refs (the mapping between refs and commits)

Plus the necessary new tables for storing a branch's own commits.
How this plugin will be used?

This plugin is an executable whose input is a git repo and may be configured to either directly write to a data store (e.g., MySQL or other databases) or a data file (e.g., parquet or other formats), depending on the environment.
This plugin will be invoked whenever target repos detect new changes, so ideally it should have an incremental mode.

Challenges

How to avoid counting the same code contribution multiple times due to rebase / cherry-pick / squash / revert / fork? #1016
How to efficiently compute and store each branch's own commits? #1011
How to make the plugin work in an incremental manner? How does it know what it has previously collected? Is the incremental approach even possible if we consider rebasing?

resolved in https://github.com/merico-dev/lake/pull/1109


Adjacent JSX elements must be wrapped in an enclosing tag

Hi there. I've been looking for a working svelte testing solution, and so far svelte-jester is broken with ESM. Then I came across this and I set this up with jest and babel with preset-env. However, it gave the following error upon running jest:
Adjacent JSX elements must be wrapped in an enclosing tag. Did you want a JSX fragment <>...</>? (10:0)

It runs fine with Vite though, so its not a issue with my code. I'm not sure how to disable this specific error.
This is my babel configuration:
{
  "presets": [
    ["@babel/preset-env", { "targets": { "node": "current" } }],
    ["@babel/preset-typescript"]
  ],
  "plugins": [
    "babel-plugin-transform-vite-meta-env",
    [
      "@babel/plugin-transform-react-jsx",
      { "runtime": "automatic", "importSource": "svelte-jsx" }
    ]
  ]
}

What’s the code?
This is likely unrelated to this package, but a difference in how Babel and esbuild handle JSX.
In particular, esbuild would probably be wrong, because JSX is a React thing, and the maintainers there treat Babel as the reference implementation.


Socket does not disconnect on hot reloading

Hi, I'm using react native tcp to implement a client connection.
However, when reloading app after i change something, the old socket is not destroyed, and lives with the new instance.
How can i fix this?

hello ! Did you fix that?


net: context: Call send_cb for TCP after ACK is received

Change the TCP send_cb call to be done after we have received
ACK to sent packet. It did not make much sense to call the
callback immediately (synchronously) after we sent the net_pkt.
Signed-off-by: Jukka Rissanen<EMAIL_ADDRESS>

No complaints on the code, but you might want to make this configurable.  This adds significant code size, won't naturally compile itself out if unused, and is only going to be useful to a small handful of apps.  I mean, it's undeniably cool, but Berkeley sockets don't actually provide a way to know when data is acked at all, and the world has survived.
An alternative way to achieve the same feature, FWIW, would be a "bytes_still_bufferd()" API or equivalent that would tell the user how far behind the current write point the latest ACK is.  Might be smaller.

Indeed the BSD sockets do not provide this kind of feature and world has survived. I have a real use case for this feature and it is related to how zephyr network stack works currently.

The HTTP server library implementation at #140 closes the TCP connection after serving the request. So after sending the response, the http code in http_server.c unrefs the connection which will send RST to client. This is just fine except that the seq/ack numbers of the sent RST are bogus because we did not wait for the ACK. The connection is terminated but the client will re-send many messages back to zephyr because of this. The motivation for this PR was to fix this packet re-send. Current code in #140 is not working properly yet in this respect as the response timeout is still 0 and code in this PR #172 is not really run by that PR yet.
Another issue with this PR that I noticed and which will require still more work, is that the received SEQ/ACK values are not yet "saved" when we call the send_cb. Any TCP messages that are sent by send_cb will have their sequence numbers wrong because of this.

Let's abandon this patch for time being. I was testing this with http-server which disconnects the client after serving the request. This connection tear-down is very fast and causes some packet resends by the client. This is probably acceptable trade off and we avoid bringing extra code that is only handling a rare use case.

The original code, if we need it later, can be found at https://github.com/jukkar/zephyr/pull/4

I didn't have a chance to look at this before, and so far just quickly skimmed thru the link posted above. I however can share following thought. There can be 2 policies for sending:

"High throughput policy": Each sending task (e.g.) thread sends data as fast and as large as possible - i.e., having 100K it allocates a send buffer, writes fitting data into it (not possible without https://github.com/zephyrproject-rtos/zephyr/pull/119), then allocates another buffer and repeats, the only thing which stops it (blocks) is unavailability of threads.
"Good citizen policy": Each sending task allocates just one buffer, fills it, sends, and waits for "sent" callback to refill and repeat.

Approach no.1 doesn't require "sent" callback at all, but a thread working like that effectively DoSes all other threads (indeed, with few threads working like that we'll likely get a deadlock in the current Zephyr). Approach no.2 requires reliably working "sent" callback, i.e. it indeed should be called once data are truly acked (or fully undeliverable), not after 1st attempt to send data out.
Consequently, I'd argue that this feature is required, though approach to implementation may vary indeed.

And - forgot to add that comment - IMHO it doesn't help to close such patches prematurely. I understand there's a concern that such may clog the queue, but some issues can't be resolved from first attempt, and IMHO it's better to keep them around (at least for few weeks) so people can think about them. We seem to have "in progress" label to keep PRs organized (or maybe can add explicit "rfc" or "wip" labels).

The PR is not lost yet and I can resubmit it after 1.8 so we can ponder this case a bit more.


Uncaught ReferenceError: template is not defined

Hello. Just installed Handlebars and trying to start working with it and I'm working according to this manual. Take a look on my code:
function ticketsToTemplate(data, element) { // calling function with two parameters: data to pass(in json format) and element to append compiled data
    let source = $('#ticket-template').html(), // parsing source template
        ticketsDOM;
    data.forEach((ticket) => { // passing json data in forEach loop
        let ticketTemplate = Handlebars.compile(source), // compiling html-template
            compiled = template(ticket); // Uncaught ReferenceError: template is not defined
        ticketsDOM += compiled;
    });
    element.append(ticketsDOM);
}

Also, please take a look on JSON and template there.
Issue Uncaught ReferenceError: template is not defined is not Googling at all, so now I'm here. Any suggestions why it could happen?
Much thanks in advance.

To blame my carelessness.


Tab colors change to match the first tab's color when dragging/reordering tabs

Azure Data Studio Version:

Version: 1.15.1 (user setup)
Commit: 83544eba2504f564d785534cc780d2073feec2cc
Date: 2020-02-15T05:03:16.545Z
VS Code: 1.42.0
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.17134

Steps to Reproduce:

Set "Sql: Tab Color Mode" to "fill" in the Settings.
Create two or more server groups with different colors.
Open a new connection to each server in new tabs.

Each tab should be filled with the server groups associated color.

Drag one of the tabs to reorder it in the window.

The tab that is being dragged changes color to match the color of the first tab in the window. Any tab that you hover over while dragging is changed to that color also.

Tangentially related, a note about tab colors and server groups:
Currently this is not reliable at all (changing connection not changing color, selecting connection from recent do not give color, etc).
In my opinion this is a critical feature, it prevents users like me doing very damaging stuff, maybe we need a way to enforce it? With an optional setting of course.
That setting would enforce a tab MUST have a color (or server group if you prefer) and if the color is not found a confirmation message should be displayed before running a query.

Do the tab colors get restored once you are done dragging?

The tab colors remain changed when you finish dragging. You actually need to click into each tab to get its color reset to the original color.

This seems to be fixed in insiders, so should be fixed in the next release. I'll track down where this fix came from for verification.

Looks like this was fixed in 1d0f76d26a45d60c54419f659a23382cbd16af46 which is a happy accident, but intentional.

Thanks for looking into this!


scikit-learn_1.3 all instance images build failed

As I use the dockerfile to build scikit_learn instance images, all intances are built successfully except 1.3 version's. It arises the Cython compiled problem as you mentioned in README. How can I solve it?
Traceback (most recent call last):
12.29     File "/opt/pyenv/versions/3.9.19/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1345, in cythonize_one_helper
12.29       return cythonize_one(*m)
12.29     File "/opt/pyenv/versions/3.9.19/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
12.29       raise CompileError(None, pyx_file)
12.29   Cython.Compiler.Errors.CompileError: sklearn/utils/_seq_dataset.pyx
12.40   warning: sklearn/utils/_cython_blas.pxd:15:18: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:17:19: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:19:15: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:21:19: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:23:15: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:25:15: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:27:15: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:29:14: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:32:15: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:35:14: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/_cython_blas.pxd:39:15: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.40   warning: sklearn/utils/arrayfuncs.pyx:34:22: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
12.46   performance hint: sklearn/tree/_tree.pyx:718:20: Exception check after calling '__pyx_fuse_6safe_realloc' will always require the GIL to be acquired.
12.46   Possible solutions:
12.46         1. Declare '__pyx_fuse_6safe_realloc' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
12.46         2. Declare any exception value explicitly for functions in pxd files.
12.46   performance hint: sklearn/tree/_tree.pyx:719:20: Exception check after calling '__pyx_fuse_4safe_realloc' will always require the GIL to be acquired.
12.46   Possible solutions:
12.46         1. Declare '__pyx_fuse_4safe_realloc' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
12.46         2. Declare any exception value explicitly for functions in pxd files.
14.48   multiprocessing.pool.RemoteTraceback:
14.48   """
14.48   Traceback (most recent call last):
14.48     File "/opt/pyenv/versions/3.9.19/lib/python3.9/multiprocessing/pool.py", line 125, in worker
14.48       result = (True, func(*args, **kwds))
14.48     File "/opt/pyenv/versions/3.9.19/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
14.48       return list(map(*args))
14.48     File "/opt/pyenv/versions/3.9.19/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1345, in cythonize_one_helper
14.48       return cythonize_one(*m)
14.48     File "/opt/pyenv/versions/3.9.19/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
14.48       raise CompileError(None, pyx_file)
14.48   Cython.Compiler.Errors.CompileError: sklearn/linear_model/_cd_fast.pyx
14.48   """
14.48
14.48   The above exception was the direct cause of the following exception:
14.48
14.48   Traceback (most recent call last):
14.48     File "", line 2, in
14.48   [ 5/64] Cythonizing sklearn/cluster/_hierarchical_fast.pyx
14.48   [14/64] Cythonizing sklearn/ensemble/_hist_gradient_boosting/_binning.pyx
14.48   [19/64] Cythonizing sklearn/ensemble/_hist_gradient_boosting/histogram.pyx
14.48   [27/64] Cythonizing sklearn/manifold/_utils.pyx
14.48   [34/64] Cythonizing sklearn/metrics/_pairwise_fast.pyx
14.48   [40/64] Cythonizing sklearn/preprocessing/_csr_polynomial_expansion.pyx
14.48   [44/64] Cythonizing sklearn/svm/_newrand.pyx
14.48   [45/64] Cythonizing sklearn/tree/_criterion.pyx
14.48   [54/64] Cythonizing sklearn/utils/_openmp_helpers.pyx
14.48   [55/64] Cythonizing sklearn/utils/_random.pyx
14.48   [57/64] Cythonizing sklearn/utils/_seq_dataset.pyx
14.48     File "", line 34, in
14.49     File "/opt/scikit-learn__scikit-learn/setup.py", line 682, in
14.49       setup_package()
14.49     File "/opt/scikit-learn__scikit-learn/setup.py", line 676, in setup_package
14.49       metadata["ext_modules"] = configure_extension_modules()
14.49     File "/opt/scikit-learn__scikit-learn/setup.py", line 610, in configure_extension_modules
14.49       return cythonize_extensions(cython_exts)
14.49     File "/opt/scikit-learn__scikit-learn/sklearn/_build_utils/init.py", line 98, in cythonize_extensions
14.49       return cythonize(
14.49     File "/opt/pyenv/versions/3.9.19/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1145, in cythonize

The issue is numpy just released to 2.0.0 recently (10 days back). For scikit 1.3, numpy isn't pinned to a specific version, so building the image causes numpy 2.0 to be pulled and scikit 1.3 doesn't build against that. If you pin numpy to 1.3 in docker/scikit-learn__scikit-learn/1.3/Dockerfile, things should work. But I'd recommend pinning all dependencies to the working version, otherwise this might repeat in the future.

Thanks for pointing it out. I got the same type of issue with all test beds now I think. The best idea I got to mitigate this is to automatically ping all dependencies to the last version at the date of the commit. But as the SWE-Bench team is working on some improvements to the evaluation setup I'm waiting for those changes first...


HTTPClient should set Authorization header for redirected requests

Motivation:
HTTPClient automatically sets Authorization HTTP header if AuthorizationProvider can locate user credentials. It is not doing that for redirected requests, however.
Modifications:
Update URLSessionHTTPClient to make use of AuthorizationProvider in redirection handler.

@swift-ci please smoke test


Develop

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

What is the current behavior? (You can also link to an open issue here)

What is the new behavior (if this is a feature change)?

Other information:

:tada: This PR is included in version 0.7.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


httptest.NewRecorder().code doesn't chance, only writes header once either on wr.write() or wr.writeHeader()

``


the highlight view is null!

怎么回事啊

hhighlight view 被回收了，可以换 addHighLight(RectF rectF, HighLight.Shape shape, int round)，自己算rectF，哪怕报错也能自己catch


fix error when run comman 'bundle install' in rails < 6.0.3

@traxanos Please check and merge this pull request. It is very important!

Is there anyone here ? Please check this pull request and merge it. It's so important.

Anyone can help me checking this pull request and merge it. Thanks so much

Anyone here ? please help !!!
@traxanos Can you help me merge this pull request?

@traxanos Do we have any update on this PR?

@traxanos Do we have any update on this PR?

Anyone can help me to merge this pull request.  Please!

Anyone can help me to merge this pull request.  Please!


DbusMethodCall: positional or keyword arguments?

The current implementation of DbusMethodCall looks like this:
@dataclass
class DbusMethodCall(Call):
    method: DbusMethod
    args: dict[str, Any] | Tuple[Any, ...] | List[Any]

in other words, it accepts the arguments (args) for the method as dict, tuple or list. The tuple and list options are currently supported by the JeepneyDbusAdapter. The dict option does not work, but crashes:
    def serialise(self, data, pos, endianness, fds=None):
        if not isinstance(data, tuple):
>           raise TypeError("Expected tuple, not {!r}".format(data))
E           TypeError: Expected tuple, not {'app_id': 'wakepy', 'toplevel_xid': 42, 'reason': 'wakelock active', 'flags': <GnomeFlag.INHIBIT_SUSPEND: 4>}

venv/lib/python3.10/site-packages/jeepney/low_level.py:199: TypeError

The crash is expected as there is no yet support for dict type of arguments for Dbus method calls.
Task
Consider, if the dict type of arguments should be supported, and if, how?

The JeepneyDbusAdapter process method implementation looks roughly like this:

class JeepneyDbusAdapter(DbusAdapter):

    def process(self, call: DbusMethodCall):
        addr = DBusAddress(
            object_path=call.method.path,
            bus_name=call.method.service,
            interface=call.method.interface,
        )

        msg = new_method_call(
            addr,
            method=call.method.name,
            signature=call.method.signature,
            body=call.args,
        )

        connection = open_dbus_connection(bus=call.method.bus)
        reply = connection.send_and_get_reply(msg, timeout=self.timeout)
        resp = unwrap_msg(reply)
        return resp

The process() gets a DbusMethodCall object (call) and creates a jeepney.low_level.Message instance (msg). The new_method_call only accepts positional arguments (the body parameter).
One option here would be to convert the call.args (which can now be dict, tuple or list) to a tuple inside the JeepneyDbusAdapter.process, since we have the available information there:
>> call.method.params
('app_id', 'toplevel_xid', 'reason', 'flags')
>>> call.args
{'app_id': 'wakepy',
 'toplevel_xid': 42,
 'reason': 'wakelock active',
 'flags': <GnomeFlag.INHIBIT_SUSPEND: 4>}

But if that would be done in the DbusAdapter subclass, then every subclass should do that. It would be more convenient if that logic was in the DbusMethodCall class. In addition, there could be some logic which asserts that the call parameter names (call.method.params) are same as the keys in the args if that is a dictionary. Perhaps even types could be checked or coerced? (at least for simple types like int, float and string?).
The DbusMethodCall could still be initialized with args being positional (tuple/list) or keywords (dict). The attribute DbusMethodCall.args could always be tuple. And in addition, there could be .get_kwargs() or .kwargs which would return the kwargs if the parameter names are given (the DbusMethod.params is optional).


Fix mangling of imports that end in "import" or "from".

Fixes #224 and #214.

Looks great! Thanks for more great work!


Add a memory game

Add a memory game in which you have to flip cards such that the pictures of that 2 card matches

You can see two cards at a time only
Plese assign me this issue under gssoc21

Please assign me this issue @ridsuteri @991rajat @Vishal-raj-1 @harshgupta20

sorry @garvitgupta13 , this project already exists in the repo , you can either try to improve that or add any other awesome project :)


Add validate_playlist functions for ToontjeHoger(Kids) experiments

In order to avoid too much manual checking of the playlists that will be delivered for ToontjeHogerKids, it would be worthwhile to add validate_playlist functions for ToontjeHoger (which ToontjeHogerKids inherits) rules files.

Duplicate of #1059


Why are snapshots stored in Bintray and JCenter

Hi,
I see that there are SNAPSHOT versions of besu library being stored in Bintray and JCenter.
HOW was it created?
Bintray UI does not allow SNAPSHOT versions to be created.

WHY was it created?
SNAPSHOT versions are not supposed to be present in the release repository.

I believe these were published using Artifactory and not the Bintray UI. Generally speaking the besu-contributors channel over on chat.hyperledger.org is best for questions like these! Closing for now.


[Transform][Fusion] support fusion for residual pattern

Compared with upstream MLIR, although we have enabled multiple consumers fusion, it furthermore need to solve following residual topology: (It seems also found in llama MLP and MHA)
op1
/  \
|  op2
|   /
op3

where the possible IR may appears like below before fusing op3:
%1:2 = scf.for() {
%2 = tiled_op1
%3 = tiled_op2
%4 = insert_slice %2
%5 = insert_slice %3
yield %4, %5
}
op3 ins(%1#1, %1#2)

Current fuse consumer only accepts single one candidate slice, saying %4, However, in fact, %5 is another slice regarding to operand of untiled op3. Thus, is is necessary to find all of the them and make sure that they are pretty matched.
BTW, this is also an interesting topic concerned by community in our upstream RFC.

Please create an issue for this PR.

Thanks for reminder! Added.


Introducing Concepts

Concepts can help reducing code complexity in templated generic programming, as well as producing simple compiler error messages. In addition, with a slightly long-term view in mind, producing a Concepts library for machine learning could become useful for the C++ community.
The goal of this task is getting hands-on and start playing with Concepts in Shogun.
As a first simple use case, we could start by moving from Traits to Concepts in the vectorized RNG pull request. For this we need simple concepts such as Integer or RealNumber. Simple concepts like these have most likely already been defined somewhere else. A place to start looking could be the range library. It would be nice to collect other tutorials and libraries for learning purposes.
Of course alternative and additional places where we could continue experimenting more are most welcome.
Detected challenges:

Compiler support. To avoid overhead during the investigation, at first I'd say is fine if this is configured somewhat a bit "hacky". For example, Concepts support could be kept in a separate experimental branch and if necessary further isolated with some build conf. flag.
...

Background:
https://www.youtube.com/watch?v=HddFGPTAmtU
https://en.wikipedia.org/wiki/Concepts_(C%2B%2B)
@saatvikshah1994
@karlnapf @vigsterkr @lisitsyn share your thoughts :-)

Playground ready #4477
I am going to try to close that PR soon merging it into a feature/concepts branch or so.


Perform accepted domain test before migrating.

The script performs a lot of saftey checks to make sure the groups are ready to go.
Recently a commenter discovered that you could get to the point where the group was created - but when the on premsies SMTP addresses would be added it would fail indicating a domain was not an accepted domain.  It is possible that the on premises organization has one or more accepted domains that are not in Exchange Online.
The script now does a pre-req check to ensure that all accepted domains that represent email addresses on the object are accepted domains in exchange online.  Current behavior if one is found broken is to abend and rely on the administrator to handle.

Fixed version 2.4.1


1.6.10 - missing migrations for hs_swat_modelinstance

@mohamedmorsy, @Castronova - Could you verify that any necessary migrations are made available to be checked into the release-candidate-1.6.10 branch for hs_swat_modelinstance.
When makemigrations is run on alpha the following is observed (using the www database).
[hydrobuild@alpha hydroshare]$ docker exec -ti hydroshare /bin/bash
root@157fbd101c0f:/home/docker/hydroshare# python manage.py makemigrations
/usr/local/lib/python2.7/site-packages/mezzanine/utils/conf.py:55: UserWarning: TIME_ZONE setting is not set, using closest match: Etc/UTC
  warn("TIME_ZONE setting is not set, using closest match: %s" % tz)
/usr/local/lib/python2.7/site-packages/django/contrib/contenttypes/models.py:161: RemovedInDjango19Warning: Model class django.contrib.contenttypes.models.ContentType doesn't declare an explicit app_label and either isn't in an application in INSTALLED_APPS or else was imported before its application was loaded. This will no longer be supported in Django 1.9.
  class ContentType(models.Model):

/usr/local/lib/python2.7/site-packages/autocomplete_light/widgets.py:19: RemovedInDjango19Warning: The django.forms.util module has been renamed. Use django.forms.utils instead.
  from django.forms.util import flatatt

/usr/local/lib/python2.7/site-packages/autocomplete_light/fields.py:8: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes.
  from django.contrib.contenttypes.generic import GenericForeignKey

System check identified some issues:

WARNINGS:
?: (mezzanine.core.W01) Please update your settings to use the TEMPLATES setting rather than the deprecated individual TEMPLATE_ settings. The latter are unsupported and correct behaviour is not guaranteed. Here's a suggestion based on on your existing configuration:

TEMPLATES = [{u'APP_DIRS': True,
              u'BACKEND': u'django.template.backends.django.DjangoTemplates',
              u'DIRS': (u'/home/docker/hydroshare/hydroshare/templates',),
              u'OPTIONS': {u'builtins': [u'mezzanine.template.loader_tags'],
                           u'context_processors': (u'django.contrib.auth.context_processors.auth',
                                                   u'django.contrib.messages.context_processors.messages',
                                                   u'django.core.context_processors.debug',
                                                   u'django.core.context_processors.i18n',
                                                   u'django.core.context_processors.static',
                                                   u'django.core.context_processors.media',
                                                   u'django.core.context_processors.request',
                                                   u'django.core.context_processors.tz',
                                                   u'mezzanine.conf.context_processors.settings',
                                                   u'mezzanine.pages.context_processors.page')}}]

?: (mezzanine.core.W02) TEMPLATE_DEBUG and DEBUG settings have different values, which may not be what you want. Mezzanine used to fix this for you, but doesn't any more. Update your settings.py to use the TEMPLATES setting to have template debugging controlled by the DEBUG setting.
hs_tools_resource.SupportedResTypes.supported_res_types: (fields.W340) null has no effect on ManyToManyField.
Migrations for 'hs_swat_modelinstance':
  0007_auto_20160428_1313.py:
    - Alter field swat_model_objectives on modelobjective
    - Alter field model_parameters on model parameter

resolved


tests: drivers: dac channel structure has a buffered parameter boolean

Add the '.buffered' field to the dac_channel_cfg structure when testing the DAC.
This boolean parameter is initialised to 'true' to PASS the test.
It follows the https://github.com/zephyrproject-rtos/zephyr/pull/57730
Fixes https://github.com/zephyrproject-rtos/zephyr/issues/58931
Signed-off-by: Francois Ramu<EMAIL_ADDRESS>

The issue does not seem to be the not enabled buffer, but incorrect output configuration. We just didn't realize this before, as an enabled buffer automatically connects the DAC to the external pin.
I think we should use LL_DAC_SetOutputConnection to fix the driver instead.

Do you mean LL_DAC_SetOutputBuffer( ... OutputBuffer ... )
plus LL_DAC_SetOutputConnection( LL_DAC_OUTPUT_CONNECT_GPIO)

Do you mean LL_DAC_SetOutputBuffer( ... OutputBuffer ... ) plus LL_DAC_SetOutputConnection( LL_DAC_OUTPUT_CONNECT_GPIO)

Yes, I think that would be the correct way. What do you think?

Do you mean LL_DAC_SetOutputBuffer( ... OutputBuffer ... ) plus LL_DAC_SetOutputConnection( LL_DAC_OUTPUT_CONNECT_GPIO)

Yes, I think that would be the correct way. What do you think?

That's better, but  initializing  the  .buffered = true in the tests/drivers/dac/dac_loopback/src/test_dac.c  in mandatory.

I have tested with nucleo_l073rz and unfortunately, it does not have the LL_DAC_SetOutputConnection and compilation fails. Looks like there are some differences between the MCU series.
Regarding the original discussion whether the output buffer is needed or not: I have checked the impedance of the DAC and the ADC in the STM32L072 datasheet.
DAC

ADC

They are quite close and without a capacitor the DAC impedance does indeed seem to be too high to drive the ADC. I have added a 100 nF capacitor and it works even without the buffer.
As we don't have the cap in standard Nucleo boards I agree to enable the buffer, now that I understand why it doesn't work without the buffer.


Fix inline checkbox example (had radio code)

PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)
[ ] If applicable, have a visual design approval

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[x] Documentation content changes
[ ] clarity.design website / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: N/A
What is the new behavior?
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

I'm not sure what I need to do to acknowledge the DCO now that I've already made the commit.  I agree!  Just trying to fix up this for you. Feel free to make a new patch yourself and delete this one if that works better.

Pulled into a new branch with #5875