How are the rules for Falco configured, or what should be added to falco in a production environment

At present, I have deployed a group of falco containers to run in the k8s cluster through helm. The current rules are not clear to me. I can only see the changes of the current node's container when I touch a file. This rule Are there any best practices in the production environment, which rules should be added, and how to modify or add rules? I am not familiar with this area, I hope I can get your help
[root@m1 deployment]# kubectl get no -o wide
NAME   STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
m1     Ready    master   20d   v1.14.3   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-1127.el7.x86_64   docker://18.9.9
m3     Ready    worker   20d   v1.14.3   <IP_ADDRESS>   <none>        CentOS Linux 7 (Core)   3.10.0-1127.el7.x86_64   docker://18.9.9

[root@m1 deployment]# kubectl get po -n kube-system |grep falco
falco-fnhqc                                    1/1     Running            0          3d
falco-hnkt9                                    1/1     Running            0          3d
[root@m1 deployment]# helm list -n kube-system
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
falco   kube-system     1               2020-10-12 11:25:50.074378454 +0800 CST deployed        falco-1.5.0     0.26.1

PTAL :point_right: https://github.com/falcosecurity/charts/tree/master/falco#loading-custom-rules

Ok,thanks, I tried the Falco rule of scripts, I commented out the values rule, by default I turned on the APiserver rule, but there seems to be a problem, no macros defined
# helm show values falcosecurity/falco > values.yaml
# ./scripts/rules2helm rules/kube1.10/rules-kube-apiserver.yaml >> custom-rules.yaml

[root@m1 rules]# cat kube1.10/rules-kube-apiserver.yaml
#
# Auto-generated set of falco rules for kube_apiserver containers
# Generated at 2018-05-23 09:58:49 UTC
#

# By default, the autogenerated rules include rules that attempt to
# restrict the set of system calls that can be performed by
# applications. However, we know that these rules are fairly FP-prone,
# so they are disabled by default. If you'd like to enable them,
# either change or override this macro's condition to "evt.num >= 0".
- macro: kube_apiserver_consider_syscalls
  condition: (evt.num < 0)

# These policies are limited to containers, specifically those where
# the container image name starts with "kube_apiserver"
- macro: app_kube_apiserver
  condition: container and container.image contains "kube-apiserver"

# Not considering all inbound networking suspect

# Not considering all outbound networking suspect

# Restricting listening ports to selected set

- list: kube_apiserver_allowed_inbound_ports_tcp
  items: [6443]

- rule: Unexpected inbound tcp connection kube_apiserver
  desc: Detect inbound traffic to kube_apiserver using tcp on a port outside of expected set
  condition: inbound and evt.rawres >= 0 and not fd.sport in (kube_apiserver_allowed_inbound_ports_tcp) and app_kube_apiserver
  output: Inbound network connection to kube_apiserver on unexpected port (command=%proc.cmdline pid=%proc.pid connection=%fd.name sport=%fd.sport user=%user.name %container.info image=%container.image)
  priority: NOTICE

# Restricting spawned processes to selected set

- list: kube_apiserver_allowed_processes
  items: ["kube-apiserver"]

- rule: Unexpected spawned process kube_apiserver
  desc: Detect a process started in a kube_apiserver container outside of an expected set
  condition: spawned_process and not proc.name in (kube_apiserver_allowed_processes) and app_kube_apiserver
  output: Unexpected process spawned in kube_apiserver container (command=%proc.cmdline pid=%proc.pid user=%user.name %container.info image=%container.image)
  priority: NOTICE

# Restricting files read or written to specific set

# Restricting syscalls to selected set

- list: kube_apiserver_allowed_syscalls
  items: [accept, connect, socket]

- rule: Unexpected syscall kube_apiserver
  desc: Detect a syscall in a kube_apiserver container outside of an expected set
  condition: kube_apiserver_consider_syscalls and not evt.type in ("<unknown>", kube_apiserver_allowed_syscalls) and app_kube_apiserver
  output: Unexpected syscall in kube_apiserver container (command=%proc.cmdline pid=%proc.pid user=%user.name syscall=%evt.type args=%evt.args %container.info image=%container.image)
  priority: NOTICE
  warn_evttypes: False

falco:
  # The location of the rules file(s). This can contain one or more paths to
  # separate rules files.
  rulesFile:
#    - /etc/falco/falco_rules.yaml
#    - /etc/falco/falco_rules.local.yaml
#    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d

I tried to export the values rule of helm and commented out the original rule. I thought that the rule would only be valid for apiserver rule at present, but the following error occurred
[root@m1 falco-extras]# helm install falco -f values.yaml -f custom-rules.yaml --set falco.jsonOutput=true falcosecurity/falco -n kube-system
NAME: falco
LAST DEPLOYED: Fri Oct 16 11:10:17 2020
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.

No further action should be required.

I'm not clear about this one. Does this default rule have to be turned on? Can't you comment other rules
[root@m1 falco-extras]# kubectl logs falco-7vs4v  -n kube-system -f
* Setting up /usr/src links from host
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc)
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc-6)
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc-5)
* Trying to load a system falco driver, if present
* Trying to find locally a prebuilt falco module for kernel 3.10.0-1127.el7.x86_64, if present
* Trying to download prebuilt module from https://dl.bintray.com/falcosecurity/driver/2aa88dcf6243982697811df4c1b484bcbe9488a2/falco_centos_3.10.0-1127.el7.x86_64_1.ko
* Download succeeded
* Success: falco module loaded
Fri Oct 16 03:10:47 2020: Falco version 0.26.1 (driver version 2aa88dcf6243982697811df4c1b484bcbe9488a2)
Fri Oct 16 03:10:47 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Oct 16 03:10:47 2020: Loading rules from file /etc/falco/rules.d/rules-kube-apiserver.yaml:
Fri Oct 16 03:10:47 2020: Runtime error: Compilation error when compiling "container and container.image contains "kube-apiserver"": Undefined macro 'container' used in filter.
---
- macro: app_kube_apiserver
  condition: container and container.image contains "kube-apiserver"
# Not considering all inbound networking suspect
# Not considering all outbound networking suspect
# Restricting listening ports to selected set
---. Exiting.

I found these 3 rules in Rules-Kube-apiserver.Yaml, I don't know if there are 3 rules. I tested the container to call SVC of K8S, and the detailed information I received through Falco is as follows
Unexpected inbound tcp connection kube_apiserver
Unexpected spawned process kube_apiserver
Unexpected syscall kube_apiserver

  "output": "09:17:28.304966461: Notice Unexpected connection to K8s API Server from container (command=node /usr/src/cicd/app.js k8s.ns=system-tce k8s.pod=tce-8b694d88c-h2vqd container=3fb74555883c image=<IP_ADDRESS>/dev-branch/xxx:20201013.095454.58 connection=<IP_ADDRESS>:46440-><IP_ADDRESS>:443) k8s.ns=system-tce k8s.pod=tce-8b694d88c-h2vqd container=3fb74555883c k8s.ns=system-tce k8s.pod=tce-8b694d88c-h2vqd container=3fb74555883c",
  "priority": "Notice",
  "rule": "Contact K8S API Server From Container",
  "time": "2020-10-16T09:17:28.304966461Z",
  "output_fields": {
    "container.id": "3fb74555883c",
    "container.image.repository": "<IP_ADDRESS>/dev-branch/xxx",
    "container.image.tag": "20201013.095454.58",
    "evt.time":<PHONE_NUMBER>304966400,
    "fd.name": "<IP_ADDRESS>:46440-><IP_ADDRESS>:443",
    "k8s.ns.name": "system-tce",
    "k8s.pod.name": "tce-8b694d88c-h2vqd",
    "proc.cmdline": "node /usr/src/cicd/app.js"
  }
}

For this to I feel to Unexpected syscall kube_apiserver, and Unexpected the inbound TCP connection kube_apiserver and Unexpected spawned process kube_apiserver both content I should how to test it, any Suggestions

This rule is generated under the default falco_rules.YAMl, which describes what conditions are generated by the rule without default falco, such as custom rules
output: Unexpected connection to K8s API Server from container (command=%proc.cmdline %container.info image=%container.image.repository:%container.image.tag connection=%fd.name)
  priority: NOTICE

The rule you mentioned is here:
https://github.com/falcosecurity/charts/blob/master/falco/rules/falco_rules.yaml#L2424-L2435

Yes, I see it, but I also provide some custom rules, how can I verify that it is valid, such as apiserver's rule mentioned above generated 3 notice events

I deployed the K8S cluster using Kubeadm, which defaulted to CenOS-7.8 and the kernel version was 3.10.0-1127.el7.x86_64
And I added all three parameters to all three of my master nodes
--audit-dynamic-configuration=true
--feature-gates=DynamicAuditing=true
--runtime-config=auditregistration.k8s.io/v1alpha1=true
The rules have been added to the POD at deployment time, as you can see in the logs, but the apiserver logs show this error

How do I turn on one of the rules in Falco_rules.YAML, many of which I find to be off

How do I turn on one of the rules in Falco_rules.YAML, many of which I find to be off

PTAL :point_right: https://github.com/falcosecurity/charts/blob/master/falco/values.yaml#L104

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Convert a couple of Sass stylesheets to vanilla CSS

For #893

🚀 Deployed on https://671f3c1ef49ca54127ac133f--alexwlchan.netlify.app


Class name changed and error handle

class name conflicting with Guzzle Client class name.
I need to use it to handle an error.

@bravocado how did you find the error?

@williamn when i'm trying to get a pokemon with id 999. Actually, the API will return {"detail": "Not found."}

@bravocado OK. Let me try to reproduce the error you found first.

@bravocado I can't reproduce the conflict error with class name. How did you run the test? Did you update the composer.json?

@williamn

Class name error

All changes on Client.php
added this new lines
use GuzzleHttp\Client;
use GuzzleHttp\Psr7;
use GuzzleHttp\Exception\ClientException;

Switch new \GuzzleHttp\Client to new Client
Guzzle already using Client name when i'm using it globally. By the way, for me it's nicer to have an unique class name.

 Still can't reproduce the error?

@bravocado if you means the 404 error, yes I find that. But I still can't reproduce the class name conflict error. Furthermore, after reading the v2 documentation I realize that the pattern of v2 is really different from v1. We can't use this existing code for v2. I will revert your changes (and mine) related to v2 and make the master only support v1. I will create a branch for working on the v2. If you would, you contribute there.


Fix typo

Github -> GitHub

@changeworld : Thanks for your contribution! The author, @wmgries, has been notified to review your proposed change.

Merging typo fix


Fix web UI errors and rename new session handler

Coverage remained the same at 50.429% when pulling 7ff088b057bf444f13a42fccbab5c2aff5e27eda on maxhq:fix-webui into 89e67a6701d6e94776c15d3b0c9d506f17d4eb08 on openxpki:develop.


[OFFICIAL] Generated Issue: Error

Please fill out the details below.
getRecentAnimeReviews api is not working.
Steps to reproduce:
hit-https://api.jikan.moe/v4/reviews/anime
Additional Details
Jikan Parser Version: v4.0.2@4df99e0df5a4adf2b92e7c6fc663de8fd4b9c325
PHP: 8.0.19
Redis: Connected
Exception: Error
Code: 0
Message: Call to undefined method Jikan\MyAnimeList\MalClient::getRecentReviews()
Trace: /var/www/jikan-rest/app/Http/Controllers/V4DB/ReviewsController.php on line 76
Request: GET /v4/reviews/anime

Thanks for your report. This is a known issue, and it has been already fixed, however we had to roll back the version of the prod environment, because of stability issues. On the Jikan discord server you can follow the latest news regarding this.

This was resolved last month.


Add grid_map utility for managing JAX parallelization/vectorization

Summary
grid_map is mapping utility for mapping a function over a "grid" of argument values. E.g. for two arrays a and b,
grid_map(f, a, b)[i, j] = f(a[i], b[j])

This works for an arbitrary number of arguments, and more generally works on JAX PyTrees using standard conventions for mapping over PyTrees (explained in the function doc string). I.e. for a and b general PyTrees with appropriate leaf shape (all leaves must have equal leading dimension size for the mapping makes sense), the above expression holds so long as we interpret
v[idx] = tree_map(lambda x: x[idx], v)

for v being any of the indexed objects in the previous expression. I.e. v[idx] is the PyTree resulting from indexing every leaf of v with idx (and we are slightly abusing notation to allow idx to be a multi-index).
Aside from the usefulness of the above form of mapping, the main point of grid_map is to offer control over how the evaluations of f get parallelized. Under the hood, it utilizes JAX's xmap to execute the mapping using a combination of device parallelization and vectorization (which they describe in their documentation as an "interpolation" between pmap and vmap). It makes natural default choices based on the device types, and a user can directly control these with optional arguments.
For Dynamics this will be of use for internally controlling parallelization in the package, whether using CPU or GPU, and generally will be useful for users to have direct access to.
Details and comments
This is currently a work in progress. This helper function was written to control parallelization/mapping in some research projects, and its design/implementation will need to be revisited for integration into Dynamics.
General to do:

Needs to be rewritten to insulate JAX imports, and raise an error if JAX isn't installed.

Design questions:

Should we add global settings that a user can specify to control "default" parallelization behaviour (max_vmap_size and devices)? The other option is to allow the user to specify these optional arguments anywhere "internal" parallelization is an option (e.g. JAX-based pulse simulation).
I recall the non_jax_argnums argument being kind of clunky - can revisit whether we want to include this.

Update (23/4/24):

It appears that the xmap has been deprecated, so the inner-workings of grid_map will need to be rethought. I think shard map may be the natural replacement.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Language Support Request: XSLT

Please could we have a beautifier for files *.xsl, *.xslt ?
Ideally, typical XML 2-space indent and a blank line between xsl elements such as xsl:template, xsl:output, etc.
Thanks!

If you change your file's grammar to XML does it work well for beautification? Let me know, because if treating this as XML works then this will be a very quick fix to get XSLT support since we can already handle XML.
To change your grammar go to Atom's menu -> Edit -> Select Grammar.

Thanks; switching the grammar allows beautify to run, and the results remain unchanged by subseequent re-beautifications, but it doesn't really add much.

correct: it generally introduces a regular 4-space indent, that's fine.

major bug: it has broken a mention of <xsl:text> by introducing spaces: <xsl : text>.

bug: it introduces a double space between elements and their attributes
<link  href="/skin/style-display.css" rel="STYLESHEET" type="text/css"/>

bug: comment PIs are not indented

nice to have: wrap+indent attributes to keep the line-length sane - currently these come out all on one line(!):

<xsl:transform
    exclude-result-prefixes="xsd xsl"
    version="1.1"
    xmlns:h="http://www.w3.org/1999/xhtml"
    xmlns:q="http://www.w3.org/2005/sparql-results#"
    xmlns:xhtml="http://www.w3.org/1999/xhtml"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

nice to have: blank lines before <xsl:template> declarations, before <xsl:stylesheet>, before <xsl:output>, and indent <xsl:param> and <xsl:variable> 4 spaces.

I'm new to Atom, but from previous lives, I used to pipe a buffer through this little shell script - included here to assist what I'd like to emulate:
#!/bin/sh

export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin

basic() {
  xmlstarlet fo -s 2
}

nltemplate() {
	sed -E 's#<xsl:template#\n  <xsl:template#g'
}

stylesheet() {
	sed -E '/<xsl:stylesheet/s# #\n    #g'
}

stylespace() {
	sed -E 's#<xsl:stylesheet#\n<xsl:stylesheet#'
}

paramspace() {
	sed -E '1,/<xsl:param/s#<xsl:(param)#\n  <xsl:\1#'
}

outputspace() {
	sed -E '1,/<xsl:output/s#<xsl:(output)#\n  <xsl:\1#'
}

variablespace() {
	sed -E '1,/<xsl:variable/s#<xsl:(variable)#\n  <xsl:\1#'
}

nowhitespaceblanklines() {
	sed -E 's#^[ \t]*$##'
}

basic | nltemplate | stylesheet | stylespace | paramspace | outputspace | variablespace | nowhitespaceblanklines

Also a couple of images just for amusement - what it gives and what I'd like.
Cheers

Interesting. Please run the Atom command from the command palette: Atom Beautify: Help Debug Editor. This will provide me with some more debugging information. Please put this into a Gist with the extension .md (it is a markdown file) and post the link here. Thanks!

https://gist.github.com/spodzone/1e15d08b795eb6214eed
The real breakage with the xsl : text happens just after a bunch of inline JS script at the bottom near the </body> tag.
HTH!

(The source used for the debug beautify already has blank lines in sensible places.)

@spodzone I provide the XML support for Atom-Beautify.  Would you mind opening a bug at https://github.com/prettydiff/prettydiff/issues and list as bullet points features you would like to extend XML beautification into a more desirable format for XSLT?


Added routes and mapping to map HTTP request

LGTM

LGTM, the datasource config could be configured later or manually in IDE


Restarting db-sync removes Postgres views used by cardano-graphql

I'm running cardano-db-sync v2.0.0 with EXTENDED=true and v1.0.0-rc.12 of cardano-graphql.
How to reproduce the problem:

start cardano-db-sync
start cardano-graphql
Everything works well, till here ✅
restart cardano-db-sync

Actual result:
Restarting cardano-db-sync removes all Postgres views used by cardano-graphql
Expected Result:
Do not break cardano-graphql.
If removal of views is necessary, maybe there's some workaround we can use to not bread cardano-graphql?
Thanks in advance for the help!!

@jaydp17 I've transferred your issue here as it's just a gap in the documentation, although it is demonstrated in code. Here's some info to help before that changes:

If removal of views is necessary, maybe there's some workaround we can use to not break cardano-graphql?

The strategy used by cardano-db-sync is shadowed here, using the hasura-cli as demonstrated in pre-start.sh. It's used within the Hasura Dockerfile, but needs to be included in a deployment definition outside of this. The only caveat at this point is if cardano-db-sync restarts, you should also restart the current cardano-graphql-hasura service. There's an effort in progress to decouple the GraphQL server from the non-outward facing service restart requirements, so I'll leave this issue open until that is complete.


[WIP] Cuboid feature user guide

This is a quick user guide for the cuboid feature.

@HollowTube , great documentation. Let's fix codacy issues (they look correct) and merge the PR. How did you create gif images?

@nmanovic
I used the Peek app on Ubuntu to record my GIFs.
The codacy should be fixed now.


listbox scheme additions (item border & category)

Currently, the color of the item selection border, and the category foreground color are hard-coded, limiting the ways in which the listbox appearance can be changed (for example, a dark background can't be used). This PR solves the issue by making those colors modifiable by the user.

Thanks for your PR


Mobile API ServiceProvider.createdAt is specified as String

Mobile API ServiceProvider.createdAt is specified as String, but returns a unix timestamp. All other createdAt fields are specified as int64 (unix timestamp). This causes issues in the mobile app. Change the OpenAPI spec to match the content and other createdAt fields.
Token.createdAt is also specified as String. Should this be int64 to?

Token.createdAt did not change recently, we already use it as a string in the app, so that should be fine.


Use https and fix some links

Given that Chrome is blocking http downloads from https sites, I thought I'd review if we were using any http links on this repo

LGTM


Support v1 tektoncd/pipeline types

Pipelines v0.43 (and LTS release 0.44) introduced the v1 API for Pipelineruns/Taskruns/Pipelines/Tasks. We should make sure Pipelines-as-code supports "handling" these resources.

not planned until we get a openshift operator wchih support v1 or our CI will be broken...

merged and working now with operator nightly on CI cluster..


update old-version-upgrade test with a k0s spec override to fix the repository issue

... of course as soon as I write this, OpenEBS moves the repo back (for the next 6 months)
so!


Products demo report an error in the mac m1

Hi @millken!
Apple has changed the NSMenu implementation in macOS Sonoma.
https://developer.apple.com/documentation/macos-release-notes/appkit-release-notes-for-macos-14
This NAppGUI check became obsolete. Fixed in this commit: https://github.com/frang75/nappgui_src/commit/197c7db6c41b8bde7e01532d4943ee95d4e8ca7e
Thanks for report!

Thanks, It works as expected.


Added grunt dev dependencies. jscs files

Tests still need to be fixed.

Argh, meant to push only the grunt dev commit


update grafaan with executor metrics

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1818  👈
#1791
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @kariy and the rest of your teammates on  Graphite


[1.10.2] Crash on post-init with EnderIO <IP_ADDRESS> installed

Issue Description:
With latest EnderIo-beta and latest The One Probe MC crashes at start.
Maybe a TOP problem? Did x-post @EnderIO.
What happens:
MC crashes while post-initializing
What you expected to happen:
Playing as normal
Steps to reproduce:
Install Forge
Install Ender IO + EnderCore
Install TOP
Play (at least try)
Crash on startup
Affected Versions (Do not use "latest"):
EnderIO: <IP_ADDRESS>
EnderCore: <IP_ADDRESS>
Minecraft: 1.10.2
Forge: 2089+ (tested on 2089 and 2092)
Your most recent log file where the issue was present:
http://pastebin.com/86ZA40i8

Confirming same crash.  Downgrading EnderIO  resolves conflict.
The One Probe: <IP_ADDRESS>.1
McJtyLib: <IP_ADDRESS>.1
EnderIO: <IP_ADDRESS>
EnderCore: <IP_ADDRESS>
Minecraft: 1.10.2
Forge: 2092
http://pastebin.com/FM0gpHAp

Workaround added for next release


config: set a longer rocksdb io limiter smooth window for raft-v2

What is changed and how it works?

Issue Number: ref #11470
What's Changed:

Set a longer rocksdb io limiter smooth window for raft-v2, this can make the compaction io through stabler. Thus, the fore ground qps can also be stabler.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List
Tests

Unit test
Manual test (add detailed scripts or steps below)

In my benchmark, after this change, the qps of sysbench oltp_write_only can be much stabler because the compaction io is stabler.

Side effects
Release note
None

/merge


Data of fusion reactor being removed when teleporting away

Modpack: Bevo's Tech Pack 2.0 (though I have noticed this in other modpacks)
I am on a Devco btp server and I have been noticing issues if you teleport away from a fusion reactor. All the data is removed from the fusion reactor (injection rate, plasma temp, energy, etc.)
Note: the fusion reactor is build in multiple chunks.
Also is there a way to set a injection rate with a redstone port?

The only was I was able to fix this was with constant loading of both chunks it is in

#3772


[CLOSED] Performant way to read context about blacklist, whitelist or keywords

 Issue by Aniketh01
Friday Apr 10, 2020 at 14:19 GMT
Originally opened as https://github.com/Aniketh01/NaWaB/issues/4

We currently have a few features like:

Blacklist: Accounts or context not to be tweeted or parsed by naWab.
Whitelist: Accounts or context which are always monitored by naWab and most of their tweets are scraped by naWab.
Keywords: We currently iterate through the list of keywords we have specific to networks that we scrap. This was done in order to generalise naWab as a scrapping tool and be used for not only network-related context but for any context.

Currently, all these are written and stored in a .txt file. We need to investigate what another sort of ways is better than to always open a file and reading the txt file for processing?
One option for us is to, use a pandas data frame to tabularise all the data and store all these into a single CSV or any other. The computation is done within the code utilising the data frame initialised once in the code is better than opening and reading a txt while throughout the process.
Is there a better way to tackle this?

 Comment by Aniketh01
Sunday Apr 12, 2020 at 09:47 GMT

As discussed,
One way to process this is using pandas, https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python. Pandas might be useful, if we scale to an extent with large precise amount of data to process.
Secondly, there is a module called pickle, not sure how things are done exactly with it. But might be worth taking a look at it.
Last, we can see if there is a way to handle this in normal sql based database.
Any other viable options we have?

 Comment by Aniketh01
Monday Apr 20, 2020 at 11:16 GMT

Fixed with #10


[Question] Merging dlls.

Could I get your permission to use ILMerge to merge your DLL into mine?
I'm making mods for a video game, and it would be really cool if I could reduce the amount of DLL files I need.

Hi, yes of course, feel free to merge MadMilkman.Ini with your DLL.
Also just in case you're interested you could achieve the same without using ILMerge, by using MadMilkman.Ini.dll as embedded resources.
Last, approach that you could also consider is to download the MadMilkman.Ini project, add it to your solution, add a project reference to it from your DLL project and then modify a bit that project reference in your ".csproj" file. This approach is described on the following link:
Combining Multiple .NET Assemblies


fix(bug): surround union alternatives with parens when they contain s…

…paces
fixes #88

So I presume this is because operator precedence in TypeScript's union types between | and something else doesn't match. What exactly doesn't match? Looking for whitespace seems like a hack that will bite us sooner or later.

The two examples I know of are
string | () => any
and
() => any []
we could search the string for the arrow operator rather than space, that
solves all the cases I know of.
Looking at the spec for Arrow Function
https://github.com/Microsoft/TypeScript/blob/master/doc/spec.md#4.11
they mention precedence briefly but I don't see a grammar definition for
all the operators and their precedence.
On Tue, Sep 8, 2015 at 4:30 PM Martin Probst<EMAIL_ADDRESS>wrote:

So I presume this is because operator precedence in TypeScript's union
types between | and something else doesn't match. What exactly doesn't
match? Looking for whitespace seems like a hack that will bite us sooner or
later.
—
Reply to this email directly or view it on GitHub
https://github.com/angular/closure-to-dts/pull/92#issuecomment-138730695
.

The grammar that also specifies operator precedence through the rule ordering is here:
https://github.com/Microsoft/TypeScript/blob/master/doc/spec.md#a-grammar
You should be able to follow down from the primary type expression to see what is legal and what needs parentheses. The Type rule itself seems to be of particular interest, note how it's branching into UnionOrIntersectionType and FunctionType at that level, and note what rules are allowed within a UnionType.

okay, I studied the grammar for a while, and I don't think I can precisely map that spec into an emitter. But, I do understand now why the ArrayType and UnionType need a PrimaryType rather than a non-PrimaryType which includes FunctionType and ConstructorType. So I think this is correct.
I added some comments explaining where these grammar productions come from, and changed to search the nested emit for the arrow (FunctionType token) or the ampersand (IntersectionType token).
If you want to comb through the grammar more precisely, then I need some help with that.

LGTMs with one docs/language nit.


default into should return Any

I was prototyping for something I might want to add to GitForge.jl
and I was using request directly.
I was, to my surprise getting a really cryptic error.
Even though AFAICT I was doing what the docstring of request said to.
And the reason was by cause I had not declared an into foor the f i was passing into request.
The safe default is for this to be Any.
Julia being a dynamic language anyway...

Looks good to me, I don't quite remember my justification for defaulting to Nothing but the JSON postprocessor works fine with Any so there's no reason not to do this.


Decrease volume

Hello,
I just built my phoniebox with HifiBerry MiniAmp, lighted buttons on GPIO and rotary control and everything works great. The only "problem" I have is that volume control works but even volume 5% is still quite loud and I'd like to decrease it - is there any possibility to do this?
Thanks
Ronny

Hi @RoBra81
welcome to the community. I remember there was a post about this somewhere, but I can not find it. What I think I remember is the following:

in the webapp you can set the maximum volume. you might also want to change the steps with which the volume can be changed and make the steps smaller
if I remember correctly, chaning the master volume with alsamixer is the way to limit the master volume for the HifiBerry card.

If you have this figured out, please add to the wiki here:
https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/MANUAL#hifiberry-dac-soundcard-details
All the best and nice ho ho holidays, micz


UE failed to connect to eNB

Hi @herlesupreeth,
I have already configure the parameter according to the info in the github and open5gs website, but the UE failed to connect to eNB.
Could you please give advice regarding this issue, and please check the following pcap file:
UE_fail_to_connect.zip
Thanks for your advice,

Looks like open5s is rejecting the attach with protocol error. I would suggest to raise an issue in open5gs repo

Hi @herlesupreeth
I've try to install open5gs & srsran on a non-docker environment and it works fine, so I guess the issue is not at open5gs.
Is there any settings that I have to add in order to be able to register the ue?
Please advice,
Thanks

I've try to install open5gs & srsran on a non-docker environment and it works fine, so I guess the issue is not at open5gs.

Then I guess you are using different versions of open5gs (check the commit id).
Also, can you please tell which gNB you are using??

I installed the Open5GS on non-docker environment using this method: https://open5gs.org/open5gs/docs/tutorial/01-your-first-lte/
How to use this version of open5gs on docker environment?
Thanks

I am not sure which commit is being used when its packaged for ubuntu. But I am guessing it when a tagged release is done.
In that case you could change the following file contents - https://github.com/herlesupreeth/docker_open5gs/blob/master/base/Dockerfile
as follows:
# Get open5gs code and install
RUN git clone --recursive https://github.com/open5gs/open5gs && cd open5gs && \
    git checkout tags/v.2.6.4 && meson build --prefix=`pwd`/install && \
    ninja -C build && cd build && ninja install && \
    mkdir -p /open5gs/install/include

Hi @herlesupreeth
Just want to confirm that the problem is solved now. Previously I skip the registration at OsmoHLR, I've only register the IMSI through Open5GS web console. Also, the code is working fine, no need to modify the dockerfile.
Thanks for your support


Fix minor regression from #169.

By setting SSL_CERTS above the if statement (for generating the
default self-signed cert), the cert will never be generated.
cc @lloesche @lingmann @kartick

Thanks good catch @brndnmtthws ... we are actually overloading the use of SSL_CERTS (earlier we set it to a boolean type, later we set it to the first cert path.) See inline code comment...

nice catch. 🐑 🇮🇹

Okay, I think this is actually what we want. PTAL.

I'm going to go ahead and merge this for now, since master is broken. We can revisit it in a separate PR if there are any follow-up comments.

Ah sorry I didn't catch it because I always tested with a custom cert. I did not mean to overwrite any previous variables. I wasn't aware of any possible --ssl-cert arg to the run file since the Dockerfile doesn't use it. In any case I will submit a PR that reverts all changes including my original commit and simply rename the variable I'm using.

I just noticed another bug with the current version which is that if --ssl-cert was provided as arg to the run file we will now supply two --ssl-cert args to marathon_lb.py because $@ is added as well.


Introduce max_http_request_size to replace max_document_size

This PR serves to consolidate ideas from https://github.com/apache/couchdb-chttpd/pull/114 (the discussion got a little long and convoluted) and to finalize implementation details.
Background
Sometimes users want to limit the actual document size of various requests. Currently, our max_document_size is a misnomer. It's actually the request body of the http request. So _bulk_docs requests with multiple docs or docs with attachments would still count within this max_document_size parameter. The name is misleading.
Requirements
This feature contains the following requirements:
R1) Use config parameter names that actually reflect their intentions.
R2) Prevent DDOS attacks.
R3) Do not allow loopholes for users to bypass the restriction and thereby create weird scenarios. We see this mostly in replication where attachments use multi-part requests.
R4) Reject a document that is actually valid. We see this scenario in replication as well. Replication needs its own section so will discuss a bit more further down.
Proposal
P1) Replace max_document_size with max_http_request_size. This initial change will serve the same purpose as before except with a different name. It meets requirements R1 and R2 above.
P2) Actually use max_document_size for update requests. So for PUT/POST requests that may or may not include an attachment, we only look at the document itself. For _bulk_docs, if a document exceeds max_document_size, we return an error with the document id, and the _bulk_docs is rejected. For update _update handlers, the same restriction is applied to update so that the updated document should not exceed the limit. This meets most of the R3 requirement, but multipart requests with replication are the problem. (More on that in replication section)
P3) Possibly introduce a use_max_document_size parameter so that we don't always execute document size computations.
Replication
The new restriction has a big impact on replication for many reasons.

Before the following fix:
https://github.com/apache/couchdb-couch-replicator/pull/49/files, replications would crash when the request limit was set too low. Now that we're changing the meaning to use actual document size instead of request size, we may have to revisit that fix.

Currently, a rare scenario exists during replication when a customer has documents very close to the old misnamed max_document_size. When we add extra query parameters or the doc is wrapped in _bulk_docs, the request size exceeds the old  max_document_size config, and replication fails. By changing the name and actually looking at the documents itself, we should be able to avoid this scenario in R4.

Replication uses multipart requests when attachments are included with the document. This means that a request is streamed. So we can't use document size as the restriction because we would have to wait for the stream to finish before extracting the document out of the entire request. @davisp suggested we read from the socket and fail once the number of bytes from the stream exceed max_document_size. However, the details are still unclear to me. I need to look at how this socket restriction will work with attachments and headers. Bottom line is that we need to get replication working such that the user can't use attachments to bypass the request, and at the same time, only the actual document is used as the limitation.

+1 to max_http_request_size though that's still a bit of a lie since we allow streaming uploads as long as we're not buffering them in RAM.
Also, I wouldn't call out replication in particular as different from any other client. We just have to make sure that all of the multipart and chunked uploads are accounted for. For instance, one of the places we'll need to add checks is here:
https://github.com/apache/couchdb-couch/blob/master/src/couch_httpd_multipart.erl#L67-L74
Granted following the code path through the various places to get to that point is less than awesome. And I think there's another similar parser to that somewhere else where we'll also need to add checks.
Also, I think you're confusing multipart/related concerns with Transfer-Encoding: chunked concerns. The multipart stuff is fine. Sometimes the JSON data even includes a header you could use to pre-reject anything. What's more difficult is chunked transfers where we don't know a content length up front, so we have to read until we exceed the max size and then abort.
Also we may want to have a discussion about what to do with replication when these sorts of configurations are different on different databases as a default policy. Ie, do we skip? Not replicate? Log loudly? Require a parameter in the replication properties that skips?

The second multipart parser is in couch_replicator_utils.erl. I don't recall why there are two. Though I  remember I was rather annoyed at the time I discovered there were two.

@davisp: You're right, I was confusing the streaming of requests with header Transfer-Encoding: chunked. Just because multipart requests are streamed doesn't mean that it used Transfer-Encoding:chunked.  It can use Transfer-Encoding:chunked, and that is where it's a little more complex. Interestingly, the special situation where we do both multipart + chunked is still broken?
https://issues.apache.org/jira/browse/COUCHDB-2295
We also have to figure out what happens when we upgrade a cluster and the document size is set to something small and now all of a sudden certain documents can't be updated because they exceed the limit.

Hmm I don't see where we are using this additional mp_parse_doc function from replicator:
https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_utils.erl#L452
I think it is dead code.

Wonder if we should use the httpd config section since max_http_request_size related more the HTTP layer than db core.

We should update following places for consistency.

couch_httpd_external.erl:67
couch_httpd.erl:454
couch_httpd.erl:580

Wonder if we should use the httpd config section since max_http_request_size related more the HTTP layer than db core.

I agree that couchdb is not the most obvious section where the one would look for it.
However due to the need to change couch_httpd there is a tough question. What section to choose:

chttpd section for both http stacks
httpd section for both http stacks
chttpd section for clustered interface and httpd section for non-clustered

I think for simplicity I'll try httpd since the setting is used by both clustered and un-clustered interface. And have 2 more prs for couch and replicator (it used in a test there).

Related PR for couch: https://github.com/apache/couchdb-couch/pull/233


Animation behaves unexpected on specific view. Animating Statusbar and subsequent views individually

In my application the the menu works great, except for one single view where i observe all elements of the view animate individually which looks really odd. I did follow the recommendation for other issues to not use "relative margin" constraints.
Please see below video to see the behavior, I'd be grateful for a hint that could explain that behavior. I can provide additional detail on the application setup if needed, but i was hoping somebody has seen such a behavior and can come up with a hint based on that video.

How do you animates these views?

Well I use custom storyboard segues of class SWRevealViewControllerSeguePushController and with animated=true
Then in the Controller of the background view I invoke
self.performSegue(withIdentifier: "activeTrip", sender: self) to switch to that respective menu element view.

Yes then 2 segues are executed (two animations), this is probably the problem.

Hm, I made sure that only one segue is triggered. In this case there is only one segue with the identifier activeTrip.
One thing to mention is that this is a segue to another storyboard, which is one of the differences over other segues. So the custom segue points to a Storyboard Reference which uses the initial ViewController of the other Storyboard.

Ok i found the reason by trial and error.
As part of the view loading process i had a section where i animate in changed constraints like this:
UIView.animate(withDuration: 1) {
   self.view.layoutIfNeeded()
}

This must have caused the double animation that you mentioned earlier.
Thanks a lot for your hint, this pointed me in the right direction.


Separate token and trivia

Maybe even introduce a fragment type for things like escapes in string?

Also remove the set-once checks from the token builder, they are kind of just useless constraints


High flake rate on web hot reload test flutter_tools/test/web.shard/hot_reload_web_test.dart

The high flake rate began around 1PM PST on Feb 3rd after this seemingly unrelated commit landed: https://github.com/flutter/flutter/commit/51b05ac7e11e1966996efa61c4d92ec904d310ab
Caused 7 of the last 11 framework commits to fail postsubmits at least once (closing the tree):

Failed engine rolls:

https://github.com/flutter/flutter/pull/120009
https://github.com/flutter/flutter/pull/119997
https://github.com/flutter/flutter/pull/119991

03:07 +9 ~4: /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart: Hot reload (index.html: Default): hot restart works without error
03:07 +9 ~4 -1: /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart: Hot reload (index.html: Default): hot restart works without error [E]
  Exception: Unexpected error response

  [{"id":2,"error":"app 'fda0f8b6-c5c7-4d51-8270-fb76b5aabc30' not found","trace":"#0      AppDomain.stop (package:flutter_tools/src/commands/daemon.dart:768:7)\n#1      Domain.handleCommand.<anonymous closure> (package:flutter_tools/src/commands/daemon.dart:273:35)\n#2      new Future.sync (dart:async/future.dart:302:31)\n#3      Domain.handleCommand (package:flutter_tools/src/commands/daemon.dart:271:5)\n#4      Daemon._handleRequest (package:flutter_tools/src/commands/daemon.dart:223:27)\n#5      _rootRunUnary (dart:async/zone.dart:1406:47)\n#6      _CustomZone.runUnary (dart:async/zone.dart:1307:19)\n#7      _CustomZone.runUnaryGuarded (dart:async/zone.dart:1216:7)\n#8      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)\n#9      _DelayedData.perform (dart:async/stream_impl.dart:515:14)\n#10     _PendingEvents.handleNext (dart:async/stream_impl.dart:620:11)\n#11     _PendingEvents.schedule.<anonymous closure> (dart:async/stream_impl.dart:591:7)\n#12     _rootRun (dart:async/zone.dart:1390:47)\n#13     _CustomZone.run (dart:async/zone.dart:1300:19)\n#14     _CustomZone.runGuarded (dart:async/zone.dart:1208:7)\n#15     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1248:23)\n#16     _rootRun (dart:async/zone.dart:1398:13)\n#17     _CustomZone.run (dart:async/zone.dart:1300:19)\n#18     _CustomZone.runGuarded (dart:async/zone.dart:1208:7)\n#19     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1248:23)\n#20     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)\n#21     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)\n#22     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:123:13)\n#23     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:193:5)\n"}]
  test/integration.shard/test_driver.dart 751:5  FlutterRunTestDriver._throwErrorResponse
  test/integration.shard/test_driver.dart 744:7  FlutterRunTestDriver._sendRequest

To run this test again: /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/dart test /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart -p vm --plain-name 'Hot reload (index.html: Default): hot restart works without error'

Not sure why this started flaking, my guess is that this issue existed all along, but the builders perhaps got slower when https://github.com/flutter/flutter/pull/119871 landed (which is when the high rate started).
I thought that hot reload doesn't work for web right now. Would it make sense to just disable these tests for now?
cc @jonahwilliams for context (it looks like you landed these tests a couple of years ago).

This is testing hot restart, its a bit misnamed. These are definitely important tests. My recommendation would be to let it burn for now and we'll deal with it on monday.

My recommendation would be to let it burn for now and we'll deal with it on monday.

SGTM. (No pressure to respond over the weekend BTW)

Not in the recipes repo, at least.

The commit where it started was https://github.com/flutter/flutter/commit/51b05ac7e11e1966996efa61c4d92ec904d310ab, which enables a bunch of new build shards. I'm not sure what else might've gotten sloshed around. (fyi @jmagman)

It sounds like in chat that this is flaking on arm64 hosts which were newly added? Can we disable the web integration tests on this shard?

This is isolated to one shard: https://ci.chromium.org/p/flutter/builders/prod/Mac web_tool_tests
I checked every instance of the failure and it only has ever happened for arm64 builds. Successful builds are a mix of arm64 and amd64.

The framework commit where it started was 51b05ac, which enables a bunch of new builders.

That commit https://github.com/flutter/flutter/pull/119871 forced the macOS benchmarks to continue to run on x64 machines (which they were doing before), and the web_tool_tests aren't happening in those builders anyway.  Much more likely this was caused by https://github.com/flutter/flutter/pull/119884 (the commit immediately before that) which tries to run non-benchmark builders on either arm64 or x64 depending on capacity.  I will make the change to force web_tool_tests to always run on x64 while this is being investigated.

LGTM. 👍
Rates since start of incident (build 10436 on builder Mac web_tool_tests):

All builds: 58% (23/40)
Excluding amd64: 74% (23/31)

https://github.com/flutter/flutter/pull/120128 has merged so this shouldn't manifest as a flake in CI anymore.  Lowering the priority since this shouldn't close the tree anymore.
This issue should continue to be investigated by the web team (@eyebrowsoffire was able to reproduce locally).

I believe this should be resolved now with a new vm_service update.

I believe this should be resolved now with a new vm_service update.

Looks like it worked!
https://ci.chromium.org/ui/p/flutter/builders/try/Mac web_tool_tests/25854/overview


Glma11 patch 3

It looks like you are attempting to push a change to the template repository, for help with deploying your own website please start a new thread in the discussions.


Style & stylesheet

When setting a font in style or a stylesheet on the frontage of the learning object, the learning object title and page title doesn't change.

It works ok for me. What css are you using?
This affects the title & header texts as well as normal page text...
body {   font: italic bold 30px serif; }
But if you're trying to change the text colour you'll need to be more specific with the selectors...
body, h1, h2 {   color: red; }

Hi Fay,
Thanks, I'll look into this.
For the Apereo Conference we can use the Xerte installation that we have
installed for us (Xerte Team). Tom will update is with the latest
changes so we have the most actual version.
I'll create the LO we use in our session there. Try to do that this week
(today//I hope).
I also create 20 demo accounts we can use in the workshops.
Bye,
Inge/
/
12Changebv
Chopinlaan 27
5242 HM Rosmalen
T<PHONE_NUMBER>
M 06-50267104
F<PHONE_NUMBER>
E<EMAIL_ADDRESS><EMAIL_ADDRESS>I www.12change.eu http://www.12change.eu
T twitter http://twitter.com/12ChangeLearn
L linkedIn http://nl.linkedin.com/pub/inge-donkervoort/7/4b/18a
FayCross schreef op 11-5-2016 10:54:

It works ok for me. What css are you using?
This affects the title & header texts as well as normal page text...
|body {
font: italic bold 30px serif;
}|
But if you're trying to change the text colour you'll need to be more
specific with the selectors...
|body, h1, h2 {
color: red;
}|
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/thexerteproject/xerteonlinetoolkits/issues/493#issuecomment-218400605

If you just want to change LO title and page title I would use something like the following:
`#x_headerBlock h1 {
font-size: 1em;
color: red;
font-family: cursive;
}
#x_headerBlock h2 {
font-size: 1.2em;
color: yellow;
font-family: cursive;
}`

Just realised I should clarify that's just for example. I would use responsive font size wrapped in media queries or not change the font siz and just change the colour/font and leave the existing responsive css to take care of the size.

Yes, Ron's css is better
(...but you should avoid using h1 or h2 in main text for page anyway as it's not great accessibility - headers used there should really start from h3)

Hi,
Can't reproduce this either anymore.
It works for me also now. Maybe a caching problem....
Bye,
Inge
12Changebv
Chopinlaan 27
5242 HM Rosmalen
T<PHONE_NUMBER>
M 06-50267104
F<PHONE_NUMBER>
E<EMAIL_ADDRESS><EMAIL_ADDRESS>I www.12change.eu http://www.12change.eu
T twitter http://twitter.com/12ChangeLearn
L linkedIn http://nl.linkedin.com/pub/inge-donkervoort/7/4b/18a
FayCross schreef op 11-5-2016 10:54:

|body {
font: italic bold 30px serif;
}|


Update documentation guidelines for contribution content

Which issue does this PR close?

Closes https://github.com/apache/datafusion/issues/12357
related to https://github.com/apache/datafusion/issues/13648

Rationale for this change
Given recent discussions on improving DataFusion stability (see https://github.com/apache/datafusion/issues/13648) I think it is time to document the criteria to add new features to DataFusion to help ensure everyone's expectations are aligned
What changes are included in this PR?

Add summary of conclusion of https://github.com/apache/datafusion/issues/12357 to the docs

Are these changes tested?
CI

Are there any user-facing changes?

Rather than state some hard rule, I went with the approach of guidance on what would need more / less discussion prior to acceptance. I felt this would give us flexibility but still given potential contributors guidance

BTW We should strive to make datafusion easier to extend. This is not only about particular APIs but also about internal design. We may be principled or not about what kind of SQL variant we provide out of the box, but we should assume extending systems provide something else (eg Ballista speaks Spark SQL, which is different than Datafusion SQL). This is obviously a teaser to longer discussion (https://github.com/apache/datafusion/issues/12723).
I approved this PR based on my understand that the wording proposed here doesn't preclude -- and actually encourages -- ground work features serving the extensibility story (the issue linked above, https://github.com/apache/datafusion/issues/12604, etc.)

BTW We should strive to make datafusion easier to extend.

I 100% agree

I approved this PR based on my understand that the wording proposed here doesn't preclude -- and actually encourages -- ground work features serving the extensibility story (the issue linked above, https://github.com/apache/datafusion/issues/12604, etc.)

Yes, this was my intention and I think it fits directly with the stated design goals (aka customizable everything):

Work “out of the box”: Provide a very fast, world class query engine with minimal setup or required configuration.
Customizable everything: All behavior should be customizable by implementing traits.
Architecturally boring 🥱: Follow industrial best practice rather than trying cutting edge, but unproven, techniques.

Should we let it hang for day-two to get other reviews?

Should we let it hang for day-two to get other reviews?

Yes I think we should leave this open for several more days to make sure anyone who wants a chance to comment can do so. Obviously we can always update the wordiing with subsequent PRs too, but I don't think there is any reason to rush this one in

LGTM

Thanks for all the comments so far. I plan to merge this tomorrow, Monday Dec 16 unless anyone else would like time to comment

🚀 📖


Revert "Scripts for Performance testing"

This reverts commit ebc07476e4a5974ebe11ef577bba0b2f9988f00c.
Signed-off-by: Daniel Widdis<EMAIL_ADDRESS>Description
Commit was not sent via PR, restoring repository to previous state
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #184 (a45da0f) into main (1aa8c5f) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #184   +/-   ##
=========================================
  Coverage     66.94%   66.94%
  Complexity      103      103
=========================================
  Files            24       24
  Lines           475      475
  Branches         25       25
=========================================
  Hits            318      318
  Misses          141      141
  Partials         16       16

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Modify Spouses Tutorial

upgraded pandas version

added torch text requirement

reformatted workshop tutorials to code refactor

downloaded data link includes train/dev/test data, external KB data, and pertained model

Another note / todo: LabelModel appears to find a spurious optimum voting all negative.  This is not surprising given the generic (accuracy-focused) loss function + class imbalance, but we should set up regularization so that this doesn't happen.
That being said, in the shorter run, can just set the seed properly (in the class initialization vs train method I think?)

@paroma TODO: Add little blurb about how we usually break ties to negative class in info extraction- can do by setting break_ties=2 in LabelModel.score- and still gets way lower than label model!

@paroma last todo: For now fix LabelModel init seed

Quick update: here are the conditional LF probs the LabelModel learns in the bad solution:
array([0.02317079, 0.01397373, 0.01203453, 0.01      , 0.01      ,
       0.06629263, 0.01      , 0.01      , 0.01      ])

So it's just the symmetry breaking issue that we expect; just need to fix the check in LabelModel for this.

Okay conclusion (after discussion with PV): We seem to have not ported the newest version of the label model, which does a simple check to resolve this final parameter sign symmetry in the correct way.  Will just port the correct version in either later tonight or after workshop, fix seed that breaks it the correct way for now.  Is also a dead simple thing to explain if someone asks ('always a fundamental sign symmetry left in these problems; in our setting we just check and pick the one that assumes LFs are better; just not in the refactor yet, but dead simple check')

done going through the comments except https://github.com/HazyResearch/snorkel/pull/1148#discussion_r296497701
and
https://github.com/HazyResearch/snorkel/pull/1148#discussion_r296496270

LGTM!  I think ready to merge in!  Last step is just confirm runs on our AWS setup, can do that when merged

Merging, apologies


Fix broken colors dependency

See: #268
colors.js was intentionally broken by it's maintainer. The solution is to fix the dependency to the latest working version.
Long term I would recommend switching to a fork of the project as more problems can be expected in the future.

@ecRobertEngel Thanks for the contribution!


Remove Kotlin as an implementation detail

Thank you for this nice library!
I was surprised to find Companion references in your README, leaking an implementation detail of your library (= using Kotlin).
But as same time you're using @JvmStatic so you're already generating functions like Light.success(...).
Why not updating the README to avoid using this ugly Companion notation in Java? ;)
Bonus: your Light.kt is only including static functions. My advise would be to remove 2 indentation levels by reformating Light.kt like this example:
@file:JvmName("Light")
package io.github.tonnyl.light

fun success(....) { ... }

Only bad point: seems not possible to have javadoc for this class :/

Hi, @pgreze ,
Thanks for your advice. I will update the readme file and remove the ugly code 😂. And I will consider the bonus.


Enhance scanOnly scenario

As of today, zAppBuild contains the capability to populate the DBB collections with the dependencies on the source level by using the option --scanOnly --fullBuild. In this case the source code is scanned for dependencies, but no outputs are created. In this context, there is a known limitation that the framework does not populate the output collection. However, the scanning process can be performed via LinkEditScanner which is included in the other build types.
We would like to enhance the zAppBuild sample framework to be able to also populate the output collection to support the scenario of migrating to Git/DBB without the need of running an initial full build when using static linkage.

Please check the below note for a short sample of this new functionality:
https://github.com/IBM/dbb-zappbuild/pull/87#issuecomment-785893455

Feature was delivered via #87 . Closing it.

@dennis-behm  Just a question on scanOnly parameter. I have executed scanOnly on my repository and created the results . I have 24 programs in buildlist.txt and build is clean but my BuildReport.html file is blank .It does not show the dependency information in HTML file and also JSON file does not have any program information. Is the BuildReport.html created only in case of FullBuild with load module or in impact build case.
How can I get dependency information in BuildReport.html so that I can view it on web browser

@dennis-behm Just a question on scanOnly parameter. I have executed scanOnly on my repository and created the results . I have 24 programs in buildlist.txt and build is clean but my BuildReport.html file is blank .It does not show the dependency information in HTML file and also JSON file does not have any program information. Is the BuildReport.html created only in case of FullBuild with load module or in impact build case. How can I get dependency information in BuildReport.html so that I can view it on web browser

The --scanOnly parameter is only scanning the files and populating the DBB collections with the dependency information. There is no build report created for this build type. You can inspect the --verbose log for dependencies or by using the DBB CLI.
Alternatively, you can run a --fullBuild --preview . See https://github.com/IBM/dbb-zappbuild/blob/main/docs/BUILD.md#perform-a-build-in-preview-mode

Hi @dennis-behm  Thanks for your response , My requirement is , I do not want to create Artifact and loads in the HLQ datasets. I want only scanning the application and generate the Buildreport with dependency information in the HTML file. Is there any other parameter available so that I can achieve this.

Hi @dennis-behm Thanks for your response , My requirement is , I do not want to create Artifact and loads in the HLQ datasets. I want only scanning the application and generate the Buildreport with dependency information in the HTML file. Is there any other parameter available so that I can achieve this.

Please read about the --fullBuild --preview. You need to understand that the BuildReport is not a complete list of the dependencies. It documents what DBB has found when resolving the dependencies. For instance it does not contain information about the object-level dependencies that are used in the impact analysis to understand the dependencies pulled in via the linkage editor.


WDS for Non-HBONE Gateways

Currently the reference from a Service or Workload to a Waypoint uses GatewayAddress.
message GatewayAddress {
  oneof destination {}
  uint32 hbone_mtls_port = 3;
  uint32 hbone_single_tls_port = 4;
}

In the waypoint capture doc we've decided that Waypoints may expose other protocols than mTLS. We need somewhere to put the desired port and protocol.
I'm pretty sure we want the GatewayAddress to capture this as it's the what we're using to join a redirected entity and the gateway. GatewayAddress.destination is similar to HTTPRoute's parentRefs[].name. We're missing a parentRefs[].sectionName equivilant to pick which of the listeners on the gateway we want.
message GatewayAddress {
  // Port exposed on the gateway. This can get mapped using Service target ports or replaced by the ztunnel's port for sandwich
  uint32 port;
  // Protocol we use on the last hop from any zTunnel to this waypoint.
  ApplicationProtocol protocol;
}

enum TunnelProtocol {
  // Uncaptured traffic
  NONE = 0;
  // HTTP CONNECT
  HBONE = 1;
  // TLS-SNI
  HTTPS = 2;
}

message Workload {
  // If set, this workload does not use zTunnel to deal with tunnel_protocol.
  bool native_tunnel;

  // The protocol we use to connect to this Workload in zTunnel outbound path.
  // Not evaluated for sandwiched Waypoints since a Waypoint may have multiple.
  // If native_tunnel is false, this _must_ be HBONE
  TunnelProtocol tunnel_protocol;
}

enum ApplicationProtocol {
 // Simply copy bytes
 UNSPECIFIED/TCP = 0;
 // Add PROXY protocol headers before copying bytes
 PROXY = 2;
}

A sandwiched waypoint has tunnel_protocol = HBONE and native_tunnel = false. It may have any application protocol. TCP would allow the "simple http" gateway case, or PROXY for more advanced waypoint implementations.
The following example demonstrates why GatewayAddress might need it's own tunnel protocol
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: hedgerow
spec:
  gatewayClassName: nginx
  listeners:
  - name: foo
    # Will match listener based on rule #1 above
    hostname: foo.hedgerow.svc.cluster.local
    port: 80
    protocol: HTTP
  - name: bar
    hostname: bar.hedgerow.svc.cluster.local
    port: 80
    protocol: HTTP

Merged in #50235


fix: build fails when -cover and/or -covermode are passed

Overhaul the Injector API to stop using decorator.Load which internally relies on the use of packages.Load (similar to go list), which caused all dependencies to be built without instrumentation as part of the instrumentation build (i.e, wasted effort).
It turns out it is viable to instead use a combination of standard library APIs to construct just the required inputs:

use go/parser.ParseFile to parse the original source code to a go/ast tree
use go/types to type-check the source code

use a custom go/importer that looks up from the -importcfg file
extract the Uses map from produced types.Info

Create a decorator.Decorator using gotypes.DecoratorResolver built from the Uses map
From this point on, the previous mode of operation is fully supported

This actually simplifies the flow quite a bit, as fewer information is now necessary from the proxy.CompileCommand (we no longer need to infer a source tree root), and allows working with files that have been transformed by other tools (such as go tool cover) seamlessly.

This PR also introduces a new way to adjust line position information (generation of //line directives) to produce more accurate results. It does so by comparing line information of nodes between the newly restored AST & the original AST, and making sure adjusted line information matches; adding new directives as necessary.

This also updates the test runners' go runtimes to go1.22.6 and go1.21.13 releases, which include bug fixes that allow us to:

forego using gotip to use the covdata tool, as it no longer keeps an excessive amount of file descriptors open;
collect coverage of the integration tests themselves during ingteration test runs.

Fixes #214

Superseded by #234, #231, #230, #229, #228, #227


Logging In With a Corporate SSL Intercept Certificate

Problem
Our corporate firewall does SSL inspection and uses an intercept certificate that is installed on all machines. In most cases, certificate errors can be resolved by adding the correct config value to the program's .rc file (i.e. ~/.npmrc, ~/.condarc, etc). That doesn't seem to work with appcenter-cli, though, or at least I haven't found the correct way to do it. So when I try to login, I see:
Error: unable to get local issuer certificate
From browsing other issues, it doesn't seem like appcenter-cli currently works behind corporate proxies/firewalls. Is that still the case?
Functional Workaround
There's probably a better way to do this, but this brute force method worked for me. What you need to do is add your cafile to the request.js library that appcenter-cli is using. I'm on a Mac, so....
$ vi /usr/local/lib/node_modules/appcenter-cli/node_modules/request/request.js
Add fs include to the top:
var fs = require('fs')
Then, add the following line to the top of the Request (options) function:
options.ca = fs.readFileSync('/path/to/your/cert.pem')
After doing that, I can at least login.

@wtollett-usgs - Thanks for documenting this workaround! Unfortunately the App Center CLI doesn't officially support making requests behind a corporate firewall, but hopefully this can help unblock some people that need this. I'll leave this issue open so that other people can find this in the future.

Since Node 7.3.0, additional certs can be made available to the node process by using NODE_EXTRA_CA_CERTS environment variable with path to the cert file. Setting this environment variable `export NODE_EXTRA_CA_CERTS="/path/to/certs/cert.pem" should work.

I'm closing this issue. Please let me know if anyone is still facing it.


Further detail

Whilst one section says a disadvantage is "customers cannot control revocation, lifecycle" etc. many customers still ask what the settings for those are (which they may need to provide for compliance).
Can that detail be added to this document, e.g. rotation, lifecycle, key specification, backup?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cd0472f9-1ed9-0154-3bfd-b25e5da64cd1
Version Independent ID: 846187d4-c24c-279f-e216-80b6507228b3
Content: Microsoft Azure Data Encryption-at-Rest
Content Source: articles/security/azure-security-encryption-atrest.md
Service: security
GitHub Login: @barclayn
Microsoft Alias: barclayn

@MrCNeale Thank you for your feedback. I have created a bug for us to consider your suggestion during the next update to this article.

#please-close


[Evented PLEG support] Do not filtering out container events with nil PodSandboxStatus

What is the problem you're trying to solve
This is a follow up action item of supporting Kubernete's Evented PLEG feature [1].
In the current implementation [2], if containerd cannot find the PodSandbox for a container upon sending container events to kubelet, it will ignore the event. This is done in this way because kubelet cannot handle nil PodSandboxStatus.
We need to update the code to make sure all events, including those with nil PodSandboxStatus, will be sent to kubelet, after kubelet can handle nil PodSandboxStatus, which is being addressed in [3].
[1] https://github.com/kubernetes/enhancements/issues/3386
[2] https://github.com/containerd/containerd/pull/7073
[3] https://github.com/kubernetes/kubernetes/pull/114351
Describe the solution you'd like
Wait for https://github.com/kubernetes/kubernetes/pull/114351 to be merged and released in k8s, and then remove the logic to ignore container events with nil PodSandboxStatus.
Additional context
No response

/assign


Add missing onDragCompleted to LongPressDraggable

When switching from Draggable to LongPressDraggable I discovered a missing callback (onDragCompleted) which can't be registered.

Oops, Good call. Thanks.
Can you add a test?

Thanks for your contribution!


Threading?

How do I create a thread in Crystal? I have code that needs to run in parallel. The source file for the Thread class says not to use it; but the alternative is not clear. Also, I am uncertain if the alternatives will actually be helpful in this case. All I want to do is normal multithreading. How is this done using Crystal?

Threads aren't supported in Crystal, and there is no plan to give direct access to them anymore. A nicer solution, like a multithreaded event loop will eventually happen (see other issues about parallelism).
Yet, Crystal is capable to run concurrent code, in an event loop. Just:
spawn do
  do_something
end

spawn do
  do_something_else
end

And you'll have 2 concurrent piece of code running.

How do I wait for the spawned process(es) to finish before continuing with my main program?

You use channels.
ch = Channel(Nil).new

spawn do
  do_something
  ch.send(nil)
end

spawn do
  do_something_else
  ch.send(nil)
end

2.times { ch.receive }

This concurrency model is very similar to that of Go, but it's still a work in progress. If in doubt, you can search how to do it in Go and then try to do the same in Crystal, though everything isn't yet supported here.

I see. Fair enough.
However, as I understand it, spawn uses Fibers - meaning only one "thread" executes at a time. If one of the "concurrent" pieces of code blocks execution, the entire program will grind to a halt. This isn't how I would define "concurrency".

In the future Fibers might be scheduled in different threads, or you might have manual control of threads. But it's still a work in progress.

Alright. So how can I fix this? https://play.crystal-lang.org/#/r/osy
Is there a non-blocking (or at least sleeping) version of accept?

accept is non-blocking, as well as any net IO. When there's a wait on an IO the runtime scheduler gives control to a new fiber, until that IO is available. The code you showed should work fine (I just tried it), but code like that won't work on play.crystal-lang.org.

It blocks the main program execution. I never get "2" printed.

sleep(1000) waits 1000 seconds

Oh. Well, I guess that might be something to do with it... ;)

Note that concurrency isn't parallelism. Concurrency is achieved by fibers, but yes, fibers won't run in parallel (but that will come).

Code inside a spawn seems to be unable to use variables declared outside the block....?

@stugol It should work. Do you have an example where it doesn't?

Also, how can I terminate a spawn from outside? Before you say "pass it a message via a channel", can I just point out that client = TCPServer.accept; client.read_line seems to be blocking.

I did have an example, but I changed it to use a channel; and now I can't reproduce it...

@stugol It is really a bad idea to terminate Fibers (or Threads, if we had them).
For example, you know, that Ruby's timeout stdlib spawns new thread with your code and terminates it from the main thread after timeout passed if it is still alive?
What happens is that you have 3 or 4 different states these threads can be in. Resulting in possible race conditions, broken database connections, production downtime (I dealt once with that, actually).
That is why other languages disallow voluntary termination of the Thread. Instead, they provide a way to interrupt thread, which effectively only sets special flag on thread, that it is politely asked to be interrupted. And it is a job of a thread in question to periodically check this flag and finish itself (by raise-ing or break/return-ing). In such languages, usually all IO and stdlib is written in a way that is aware of this behavior and acts correctly, when thread is asked to terminate. That is why you usually don't have to deal with it, unless you are implementor of low-level library.

follow up to my previous comment:
I am not sure, that we don't have such a functionality here in Crystal. And I think it is a good idea to have it once we start working on concurrency and parallelism with more focus.

AFAIK, Go's take on this is to use select { case ... } construct to listen on multiple IO and channels simultaneously, thus providing a way to simultaneously block on IO and wait for message on channel, thus providing an elegant way to finish a goroutine, when asked. I am not sure what happens to the IO operation in question after that, though.

I think that true parallelism would be a big selling feature for Crystal. It's something that's definitely lacking from Ruby and other languages.

@mikejholly The last time this was brought up, I think someone mentioned that parallelism will be added eventually, but I don't think it will expose threads (a la C/C++) since those will be just an implementation detail for how parallelism works; it will probably be abstracted away, in a similar way how the spawn or parallel (this just uses spawn, btw) macros do now.

@omninonsense Personally, I think Threads should be available BUT the nice abstractions like spawn and parallel should be preferred. I think Clojure does this well.

So if the only way to achieve something (listening on a port, reading user input etc) happens to be a blocking call - or long-running loop - my entire program breaks? That doesn't sound useful.

@stugol, If you do a blocking call inside of a Fiber, all other Fibers will
continue their execution normally, because scheduler will put blocked Fiber
to sleep until blocking call is completed.
--
Best Regards,
Oleksii Fedorov,
Sr Ruby, Clojure, Crystal, Golang Developer,
Microservices Backend Engineer,
+49 15757 486 476
On Thu, Dec 24, 2015 at 8:16 PM, stugol<EMAIL_ADDRESS>wrote:

So if the only way to achieve something (listening on a port, reading user
input etc) happens to be a blocking call - or long-running loop - my entire
program breaks? That doesn't sound useful.
—
Reply to this email directly or view it on GitHub
https://github.com/manastech/crystal/issues/1967#issuecomment-167150982.

No, that doesn't make sense. Consider this pseudocode:
def GetInput
   do
      accept keyboard input
   until [user has pressed enter]
end

Fiber 1 {
   GetInput()
}

Fiber 2 {
   [some other long-running task]
}

Granted, spinwaiting isn't a very good way to go about things, but the point is, GetInput is a blocking call. Surely the second fiber can't proceed until the user presses the enter key? Fiber-based concurrency generally requires polite sleep()ing from all participants; and if one fiber misbehaves, the entire program is up the swanny.
Windows 3.1 used to work like that. All modern OSs use preemptive multitasking, and there's a good reason for it.
Sure, fibers are really awesome. But there are cases where they're not appropriate; and in such cases, it would appear that Crystal doesn't offer any alternatives at all. I'm actually considering switching to Google Go for this project; simply because I anticipate a lot of concurrency, and don't want to be utterly boned when I eventually come across a blocking call I cannot avoid using.

read a character from the keyboard - is a non-blocking IO, at least it should be. Therefore it will trigger proper re-scheduling for other fibers.
If you do not like, how it is implemented here, in Crystal, you will be disappointed in Go's concurrency, because it is the same. Just rename Fiber => Goroutine.

@stugol I might be wrong, but it looks like you are mixing 2 different concepts here: concurrency and parallelism.
They are connected, but very different. What you see here in Crystal (and in Go) is pure implementation of concurrent model (called CSP), and it is actually complete.
Threads - is just one way to do parallelism. Very poor way at that.

In Go, parallelism is achieved by telling go program to run over N cores instead of one. In that case, sometimes some goroutines will actually run on different cores at the same time. This type of parallelism will not change your coding style at all, and your program should not be different, if it is run on 1 core or N. This has one limitation - max value of N is determined by amount of CPU threads (usually cores * 2).

From Go blog on the topic: http://blog.golang.org/concurrency-is-not-parallelism - "Concurrency is not parallelism"

Parallelism make sense only for number crunch tasks not IO operations. In go if you really need do parallel math you shouldn't run multiple copies, but limiting this computations with buffered channels where channel capacity = number of CPU cores.

"Goroutines are multiplexed as needed onto system threads. When a goroutine executes a blocking system call, no other goroutine is blocked."
The same cannot be said for Crystal, as I understand it.
If I need to perform parallel number crunching - or if, for some reason, an IO call is blocking and there's nothing I can do about it - how can I solve this in Crystal?

As far as such rare case goes, I suppose you will have to wait until the language designers come up with solution for that. They are still in progress of designing concurrency and parallelism at the moment as far as I know.
I would expect that it will be similar to go's approach, i.e. ability to parallelize fibers on cores and buffered channels with size of number of cores. Though it is my speculation, assumption.
If you need something right now, I would recommend actually using Go. In my opinion concurrency model in Go is very good one.
BTW to my knowledge it is discouraged to try to contribute to this part of the language at the moment, before there is high-level vision for this.
Alternative solution to this problem is to create standalone library for threads or for parallelizing of fibers. Probably it will use some C library. It is a nice idea, since in the progress one would learn a lot of internals and overall how it works and should work.
Von meinem iPhone gesendet

Am 25.12.2015 um 3:47 PM schrieb stugol<EMAIL_ADDRESS>"Goroutines are multiplexed as needed onto system threads. When a goroutine executes a blocking system call, no other goroutine is blocked."
The same cannot be said for Crystal, as I understand it.
If I need to perform parallel number crunching - or if, for some reason, an IO call is blocking and there's nothing I can do about it - how can I solve this in Crystal?
—
Reply to this email directly or view it on GitHub.

Goroutines are multiplexed as needed onto system threads. When a goroutine executes a blocking system call, no other goroutine is blocked

I believe such system calls are executed on separate thread(s). So they will still not block other fibers. Am I correct or not @asterite?
Von meinem iPhone gesendet

Am 25.12.2015 um 3:47 PM schrieb stugol<EMAIL_ADDRESS>Goroutines are multiplexed as needed onto system threads. When a goroutine executes a blocking system call, no other goroutine is blocked

Is it possible, using Crystal, to make API calls to create threads? For example (pseudocode):
def threaded_fn
   ...long-running blocking operations...
end
thread = WinAPI.CreateThread(address-of(threaded_fn))

Obviously I wouldn't expect you guys to necessarily be experts on the Windows API, it's just an example. In C, I can call CreateThread (or the Linux equivalent) and it returns a handle. I would need to obtain the address of threaded_fn in a manner compatible with C. Can this be done?
In essence, how do I make a C-style function pointer to a Crystal function? For example, CreateThread expects a function pointer with the signature unsigned long __stdcall (*fn)(void*). So presumably I would need def threaded_fn(param) and a way to make a function pointer from it. I'm a bit hazy on how to do that.
I don't mind using Crystal's built-in concurrency support; but I need the option to use real threads if the need arises.

Hm. Re-reading the documentation, it seems that I simply specify - when importing the C call - that it takes a callback. So:
@[Link("kernel32")]
lib Kernel32
   fun CreateThread(attrs : SECURITY_ATTRIBUTES*, stacksize : SIZE_T, callback : void* -> UInt32, param : void*, creation_flags : UInt32, thread_id : UInt32*)
end

def threaded_fn(param : void*)
end

thread_handle = Kernel32.CreateThread(nil, 0, threaded_fn, nil, 0);

Or something to that effect?
(API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682453(v=vs.85).aspx)

In which case, what happens when I deploy the program on a non-Windows platform? If I simply refrain from calling Kernel32 functions (and call others instead, obviously), will the lib declaration simply be ignored? Or will an error be raised because kernel32.dll doesn't exist?

Does Crystal actually work on Windows right now? I would think you'd have to just use pthreads or something...

What's a pthread?

Actually, it's a Posix thing...

Isn't that kinda the same thing? ;)

No; Posix also works on OSX, BSD, and other Unix-y systems.

I've managed to achieve threading using API calls. It wasn't easy, but it works; and I can pass in arbitrary data. Example:
thread1 = threaded {
	DoWork1()
}
thread2 = threaded(27) { |n|
	DoWork2(n)
}
thread1.join
thread2.join

The magic:
@[Link(ldflags: "-pthread")]
lib ThreadLib
	type Thread_t = UInt64
	type Thread_attr_t = Int64
	fun pthread_create(lpThread_id : Thread_t*, attr : Thread_attr_t*, callback : Void* -> Void*, arg : Void*) : Int64
	fun pthread_join(thread_id : Thread_t, value_ptr : Void**) : Int64
end

class ThreadException < Exception; end

def runcode(arg, &block); yield arg; end
def runcode(&block); yield; end
macro threaded(&block)
	%success = ThreadLib.pthread_create(out %thread_id, nil, ->(%param : Void*) {
		runcode {{block}}
		return Pointer(Void).null
	}, Pointer(Void).null)
	raise ThreadException.new("Could not create thread") unless %success == 0
	ThreadMgr.new %thread_id
end
macro threaded(param, &block)
	%success = ThreadLib.pthread_create(out %thread_id, nil, ->(%param) {
		runcode (%param as typeof({{param}})*).value {{block}}
		return Pointer(Void).null
	}, pointerof({{param.id}}) as Void*)
	raise ThreadException.new("Could not create thread") unless %success == 0
	ThreadMgr.new %thread_id
end

class ThreadMgr
	def initialize(thread_id)
		@thread_id = thread_id
	end
	def join
		ThreadLib.pthread_join(@thread_id, nil)
	end
end

Unfortunately, loads of built-in Crystal functions are not thread-safe - indeed, they don't even work in native threads. For example, any attempt to puts to the console runs the risk of a could not get the current fiber exception from IO::FileDescriptor#wait_writable...
I guess this solution will work for computation and such; but large parts of Crystal will simply break if I try to thread them. The question is: Should I persevere with Crystal on this project, or switch to Go instead? I'm trying to write a backup server, and it has to be reliable. And more to the point, I don't want to get halfway finished and suddenly find myself unable to proceed without proper parallelism support.
For example, when a client connects, there will be a massive amount of checksumming and other math-related stuff happening; and I don't want to block any other work. How would I write a long-running mathematically-intensive computation to play nicely with other fibers?

You could just do thr = Thread.new { ... } and have a thread running. It works, but its unsupported, and the thread will crash immediately on any IO calls, because the event loop can't be found for the current thread. Actually, even a sleep 1 will crash the thread.
Again: coroutines will be multiplexed across a pool of threads (or something like that), so a CPU bound coroutine won't block other coroutines until it's done, but this is not possible, now.

I think it is a bit early to use Crystal for something that "has to be
reliable", especially if you see such a big blocker on the road already.
Yes, some companies are actually using Crystal in production already,
basically, they acknowledged all risks, that this involves and decided that
for a certain parts of their product it is fine to use non-production-ready
language. (They do it, because they are sure, that they can rewrite one of
these parts in the matter of hours if the need arises).
So my advice here would be - use Go.
If you want to use Crystal right now - try splitting the server in multiple
independent communicating parts: one or two in Go, where you care about
parallelization, everything else in Crystal; communicating through a socket
or something else.
--
Best Regards,
Oleksii Fedorov,
Sr Ruby, Clojure, Crystal, Golang Developer,
Microservices Backend Engineer,
+49 15757 486 476
On Fri, Dec 25, 2015 at 9:33 PM, stugol<EMAIL_ADDRESS>wrote:

I've managed to achieve threading using API calls. It wasn't easy, but it
works; and I can pass in arbitrary data. Example:
thread1 = threaded {
DoWork1()
}
thread2 = threaded(27) { |n|
DoWork2(n)
}
thread1.join
thread2.join
The magic:
@[Link(ldflags: "-pthread")]
lib ThreadLib
type Thread_t = UInt64
type Thread_attr_t = Int64
fun pthread_create(lpThread_id : Thread_t*, attr : Thread_attr_t*, callback : Void* -> Void*, arg : Void*) : Int64
fun pthread_join(thread_id : Thread_t, value_ptr : Void**) : Int64
end
class ThreadException < Exception; end
def runcode(arg, &block); yield arg; end
def runcode(&block); yield; end
macro threaded(&block)
%success = ThreadLib.pthread_create(out %thread_id, nil, ->(%param : Void*) {
runcode {{block}}
return Pointer(Void).null
}, Pointer(Void).null)
raise ThreadException.new("Could not create thread") unless %success == 0
ThreadMgr.new %thread_id
end
macro threaded(param, &block)
%success = ThreadLib.pthread_create(out %thread_id, nil, ->(%param) {
runcode (%param as typeof({{param}})).value {{block}}
return Pointer(Void).null
}, pointerof({{param.id}}) as Void)
raise ThreadException.new("Could not create thread") unless %success == 0
ThreadMgr.new %thread_id
end
class ThreadMgr
def initialize(thread_id)
@thread_id = thread_id
end
def join
ThreadLib.pthread_join(@thread_id, nil)
end
end
Unfortunately, loads of built-in Crystal functions are not thread-safe -
indeed, they don't even work in native threads. For example, any
attempt to puts to the console runs the risk of a could not get the
current fiber exception from IO::FileDescriptor#wait_writable...
I guess this solution will work for computation and such; but large parts
of Crystal will simply break if I try to thread them. The question is:
Should I persevere with Crystal on this project, or switch to Go instead?
I'm trying to write a backup server, and it has to be reliable. And more to
the point, I don't want to get halfway finished and suddenly find myself
unable to proceed without proper parallelism support.
For example, when a client connects, there will be a massive amount of
checksumming and other math-related stuff happening; and I don't want to
block any other work. How would I write a long-running
mathematically-intensive computation to play nicely with other fibers?

—
Reply to this email directly or view it on GitHub
https://github.com/manastech/crystal/issues/1967#issuecomment-167261460.

You can't compute in threads, because you can't communicate back the result, or even just ask the question, since IO, Channel, etc are broken in threads.
A solution to your problem, now, could be to fork { ... } and talk through an IO.pipe or a UNIXSocket for example.

@ysbaddaden: Regarding Thread.new, presumably I would have to use the sleep kernel function, rather than the Crystal one. But in any event, there would be quite a large number of things I couldn't do - IO, for example, or thread synchronisation. So maybe it's just not feasible to use threads at this time, even thought it can actually be done.
fork might be the answer, but I'm a bit uncertain how it works. Does it literally create two processes, just as if I ran the program twice to begin with? I guess I'd create the IO.pipe first, and both processes would then have access to it because it was created before the fork took place?
I don't have much experience with pipes - other than the type you use in a terminal, that is. Do you have an example of usage?
@waterlink: To write the program in two different languages, presumably I would have to produce two separate executables, run them both, and somehow have them talk to each other?  If one part could load the other part as a DLL/SO or something, then maybe it would be a more sane plan; but afaik this is not possible.

@ysbaddaden: Wouldn't fork open a can of worms though? I'd end up with potentially half a dozen copies of the executable running. If any of them crashed or were terminated, it would impact the master program which would continue waiting for messages from the dead process. Also, by definition, these forks would be running blocking code (else I could just use fibers), so communicating with them (or terminating them) in a timely manner would be difficult.
Then again, I guess I'd have similar problems with threads.
I guess the most important question is, then: How do I ensure that a fiber plays nice with others?
For example (pseudocode):
fiber {
   while [condition]
      [some complex maths]
   end
}

Do I just stick a sleep(0) call in the loop? Or is there a better way to avoid blocking other fibers?

@stugol sleep(0) in your example should be the best bet.
On 2 languages, you are totally right, that is why I told If you want to use Crystal right now, meaning like "badly want to use it and go out of your way to do something complex". On the other hand it could be a very interesting experience :)

You may Scheduler.yield (I think), but this may be internal/private API, I'm not sure.

@waterlink having looked into it further, I'm afraid Go isn't going to do. It lacks overloading AND generics, meaning it's essentially useless. You need at least one of those (IMO) to get anything done at all. Opinions aside, it completely violates the DRY principle, as can be seen here.
It also lacks nullable variables, extension methods and a ternary operator. You can't even use an if as a ternary:
result = if a then b else c     // error!

Pish!

@stugol Haha, that is just wonderful how you discovered it so quickly. I can only agree. Though it does have good concurrency model; honestly, concurrency model is the only thing I like in Go. So if you need very fast, concurrent, parallelizable simple application, then Go is pretty decent choice. If you anticipate a bigger codebase size, or more rapid change of the codebase, it is not the best choice. You can try Scala with Akka actors (or Akka streams) in this case, I suppose. Or you could, again, make 2 applications, one that is critical to be fast and parallelizable (in any language of your choice that allows to make this job done, not necessary the language of your liking) and other part is for everything else in a language you like; making them communicate correctly is a good exercise :)

@stugol Try Felix. It's horribly documented, but it's quite fast and has a concurrency model similar to Go. The mailing list is pretty helpful, though.

@kirbyfan64 Wow: Supports both ML-like functional and C-like imperative programming styles.

@kirbyfan64 Thanks to you I have put a new awesome item on my todo list :)

@kirbyfan64 But how production-ready is it? It doesn't say.

Hm. Seems it's been in production for 15 years. So maybe it's ready ;)

Hmm, releases page looks really strange on github: https://github.com/felix-lang/felix/releases

Yeah, only one release. That's a bit weird.

Another one to look at is Rust. The same concurrency model and very good type system. Not saying about memory usage compile time safety.

Already considering Rust. It doesn't support overloading (boo!) but might be usable.

@stugol Might be an interesting read on the topic<EMAIL_ADDRESS>

Well, it's sort-of production-ready. As for the one release, it's that the release systems was changed to GitHub about a month ago.

And the web server is self-hosting.

Self-hosting? What does that mean?
I just checked out the git repo and tried to build it. The GNUmakefile is a bit buggy, and doesn't work. I'm having to monkeypatch it.

And now I'm getting:
Dynamic linkage error
filename: toolchain_gcc_linux.so
operation: LoadLibrary/dlopen
what: Cannot find dll/shared library
GNUmakefile:174: recipe for target 'rtl' failed
make: *** [rtl] Error 253

So I guess it's not production-ready then.

Considering multiple people have built that multiple times, that's kind of...weird...
Try the mailing list.

@stugol Could you use the Google Groups for this kind of discussions? Issues need to be eventually closed, but these discussions don't seem to have an end.

Reading this issue comments gives knowledge!

Excuse me @asterite . But crystal in 0.24.2 have automatic multithreading? Code and proof on screenshot

@foi The GC is multithreaded

Enabling Crystal’s New Multicore Support
https://dev.to/jgaskins/enabling-crystal-s-new-multicore-support-4l4g


Expose is_keyword, is_vector, is_map, etc on edn::Value

Over in #196, @rnewman wants some helper functions to make destructuring edn::Value more pleasant.  This ticket tracks adding is_* predicates for all the variants, and potentially as_vector(Value) -> Option<Vec> and as_map(Value) -> Option<BTreeMap<Value, Value>, etc.
We should aim to provide the exact same helper API as serde json library.

Source pointer:
https://github.com/mozilla/mentat/blob/rust/edn/src/symbols.rs