nodejs 18 docker image

I have cmake installed on the host my nodejs application needs to build binding files via cmake how do i add cmkae to the docker image? FYI i have no experience with docker.

also tired using your image as a base but it fails to instal cmake
Use the existing image as the base image
FROM quay.io/yajtpg/pterodactyl-images:nodejs-18
RUN apt-get update && apt-get install -y cmake
also tried RUN apk add cmake


.travis.yml needs an update

I think the Go developers aren't going to maintain the "release" tag anymore.  (Git might not handle moving a tag to a new commit the way Mercurial did.)  So I think we'll have to maintain two numbered releases (which currently should be 1.3.3 and 1.4.1) in addition to tip.
update:  Oh wait, I see we weren't even using "release".  And it looks like gvm is smart enough to take the latest point release.  Anyway, it's time for a version update.  should be 1.3 and 1.4

I pushed it to a branch to give travis a chance to build it before it goes to master (in case I screwed it up).

Ah thanks.  I know, almost silly to file an issue but I had thought of this and then forgotten a couple of times already.  I was busy today when I was reminded of it again and didn't want to take time to mess with it so I just filed the issue.

hm :/  I don't understand...  Now it looks like that didn't pull the point releases.  Do we have to say 1.3.3 and 1.4.1?

Maybe we do, I'll try that!

Yeah, this isn't our fault.  Travis has changed they way they select versions.  Compare the last two build logs.  They were using "gvm" and are now using "gimme".  So maybe before gvm would pick the latest point release but now it has to be specified explicity for gimme.

Cool, I'll update the PR

LGTM

Fixed with PR https://github.com/exercism/xgo/pull/132


Add type guards

This refactors the code and also helps mypy not to complain as much
See the related PEP

type guard does not work for class property is_matched for now


Login error

I have this error when I try to log in my Microsoft account :
{"reason":"{\n  \"path\" : \"\/minecraft\/profile\",\n  \"error\" : \"NOT_FOUND\",\n  \"errorMessage\" : \"Not Found\"\n}","error":true}

Do you use Game Pass?


Fixed setting of DNS_CLUSTER_IP in bootstrap.sh

Description of changes:

Added route table lookup to determine interface handling default route
Replaced call to metadata service with cat of
/sys/class/net/$DEFAULT_ROUTE_INTERFACE/address, since the data is
available on-instance
Restored separate setting of CIDRS and TEN_RANGE variables for easier
debugging
Use tr to translate any spaces to new lines in CIDRS variable

Issue #, if available:
https://github.com/awslabs/amazon-eks-ami/issues/222
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I just checked on my laptop and while connected to both ethernet and wifi, I have two default interfaces:
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        <IP_ADDRESS>         UG    100    0        0 enp0s31f6
default         _gateway        <IP_ADDRESS>         UG    600    0        0 wlp3s0

$ route | grep '^default' | grep -o '[^ ]*$'
enp0s31f6
wlp3s0

Could this ever happen on an EC2 instance? So many gotchas to consider!
Adding head -n 1 as was originally in the code would resolve that:
$ route | grep '^default' | head -n 1 | grep -o '[^ ]*$'
enp0s31f6

As would setting the --max-count flag for grep:
$ route | grep -m 1 '^default' | grep -o '[^ ]*$'
enp0s31f6

@drewhemm with two default routes you what the one with the lowest metric.
One approach to to ask where there server would route to, for a non-local address, and then take the interface it would use.
$ ip route get <IP_ADDRESS>
none <IP_ADDRESS> via <IP_ADDRESS> dev eth0 proto unspec src <IP_ADDRESS>

So the interface is:
$ ip route get <IP_ADDRESS> | grep -oP 'dev \K\S+'
eth0

@whereisaaron I'm afraid that doesn't work; on both my laptop and an EC2 instance, the default route has a gateway other than <IP_ADDRESS>, so that command returns the loopback device:
$ ip route get <IP_ADDRESS>
local <IP_ADDRESS> dev lo src <IP_ADDRESS> uid 1000
    cache <local>

My laptop:
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        <IP_ADDRESS>         UG    100    0        0 enp0s31f6
default         _gateway        <IP_ADDRESS>         UG    600    0        0 wlp3s0

EC2 instance:
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         <IP_ADDRESS>         UG    0      0        0 eth0

But, looking at the route table, my two default interfaces have different metrics and the one with the lowest metric is on top. I need to do some research to see if that is coincidental or predictable.

Confirmed table ordering is by metric by changing the metric:
Compare
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         dsldevice.lan   <IP_ADDRESS>         UG    600    0        0 wlp3s0
default         <IP_ADDRESS>         <IP_ADDRESS>         U     1000   0        0 enp0s31f6

with
route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         <IP_ADDRESS>         <IP_ADDRESS>         U     100    0        0 enp0s31f6
default         dsldevice.lan   <IP_ADDRESS>         UG    600    0        0 wlp3s0

So we can take the first default entry in the route table. @mogren Are you happy for me to update this PR accordingly?

Oh thanks @drewhemm, my mistake for using the special case '<IP_ADDRESS>' address as the target/test address 🤦♂. Just use any normal test address. Here is a test on an EC2 instance:
$ ip route get <IP_ADDRESS>
local <IP_ADDRESS> dev lo  src <IP_ADDRESS>
    cache <local>
$ ip route get <IP_ADDRESS>
<IP_ADDRESS> via <IP_ADDRESS> dev eth0  src <IP_ADDRESS>
    cache
$ ip route get <IP_ADDRESS>
<IP_ADDRESS> via <IP_ADDRESS> dev eth0  src <IP_ADDRESS>
    cache

This can't really not work, since you are asking the kernel, 'hey were would route default traffic, given your route table`.
I'd say asking the kernel is technically more reliable than scraping the route table output. However, I image scraping the route table will also be 100% effective. 😄

I quite like the idea of using ip route get, as long as we are happy to hard code an external IP address like <IP_ADDRESS>; does that add any kind of dependency on that IP address being up and reachable?

@drewhemm no the target IP doesn't even have to exist or be accessible, it is just a routing decision, no packets are sent. The gotcha would be if there was a separate, more specific, routing table entry that matched the test address, e.g. if there was a route table entry to send all <IP_ADDRESS>/16 addresses via eth1.
Whatever you decide, it might be better to base it around the iproute2 ip route command rather that the deprecated iproute route command. Apart from not being deprecated, the ip output is generally more script friendly.
Also, it is probably not ideal to rely on the sort order of the route human-friendly output, as I don't see anything in the man page that guarantees the sort order. Better to sort yourself and be sure.
If you want to use your suggested method to grep for lowest-metric route. You can use ip route show default and sort/sed/head/grep that output. Something like:
ip route show default | sort -nk 9 | head -1 | grep -oP 'dev \K\S+'

That does still assume the number and order of columns, but you can remove that assumption and pick out the name+value pairs you want (in any order in the output), to get something that should be quite robust.
ip route show default | sed -nre 's/.*(^|\s)((dev|metric) \w+).*\s((dev|metric) \w+).*/\2 \4/p' | sort -nk 3 | head -1 | grep -oP 'dev \K\S+'

Hi @whereisaaron, I tested those two options on an EC2 instance: the first works but the second returns no output...

@drewhemm looks like Amazon Linux has a real old version of the ip utlity. Although you set the metric when adding route, there appears no way to display the metric when listing routes!! It also doesn't support 'default', you have to use '<IP_ADDRESS>/0'.
$ ip -V
ip utility, iproute2-ss160111
$ ip --details route
unicast default via <IP_ADDRESS> dev eth0  proto boot  scope global
unicast <IP_ADDRESS> dev eth0  proto boot  scope global
unicast <IP_ADDRESS>/20 dev eth1  proto kernel  scope link  src <IP_ADDRESS>
unicast <IP_ADDRESS>/16 via <IP_ADDRESS> dev eth1  proto boot  scope global

Compare with:
$ ip -V
ip utility, iproute2-ss180129
$ ip --details route
none default via <IP_ADDRESS> dev eth0 proto unspec scope global metric 0
none <IP_ADDRESS>/24 dev eth0 proto unspec scope global metric 256
none <IP_ADDRESS> dev eth0 proto unspec scope global metric 256

So with that old ip utility is looks like the best you can do is get the first listed default device. This means you aren't sorting by metric and counting on the output order 😞
ip route show <IP_ADDRESS>/0 | head -1 | grep -oP 'dev \K\S+'

@whereisaaron That works on both the EC2 instance and my machine, so I am happy with that.
Although it looks like the metric isn't displayed with the older version of ip route, it merely doesn't show the metric if it is zero. I confirmed this by adding two more routes:
$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
<IP_ADDRESS>         <IP_ADDRESS>     <IP_ADDRESS>         UG    0      0        0 eth0
<IP_ADDRESS>         <IP_ADDRESS>         <IP_ADDRESS>         U     5000   0        0 enid4de56097eb
<IP_ADDRESS>         <IP_ADDRESS>         <IP_ADDRESS>         U     6000   0        0 eni00c29516e4a

$ ip route show <IP_ADDRESS>/0
default via <IP_ADDRESS> dev eth0
default dev enid4de56097eb scope link metric 5000
default dev eni00c29516e4a scope link metric 6000

The route ordering is still metric-based, again confirmed by altering the metrics of the interfaces, so taking the first one is a solid approach.
I've updated my PR accordingly.

Is something holding this up from being merged?

Thanks for the detailed writeup and so sorry for the delayed response.
While this does side-step a possible 404 on the http://<IP_ADDRESS>/latest/meta-data/network/interfaces/macs/ endpoint, there is no guarantee that the http://<IP_ADDRESS>/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks endpoint will not 404 either. What do you think of adding a bootstrap argument where you can set the CIDR blocks? That way you could bypass the metadata calls for this case.

Yeah, I think you're right as I am still occasionally seeing nodes with the wrong cluster DNS. A bootstrap argument should be bulletproof. I will update the PR.

What do you think of adding a bootstrap argument where you can set the CIDR blocks? That way you could bypass the metadata calls for this case.

I've amended the PR to use an argument instead, with the default value of <IP_ADDRESS> unchanged. However, this introduces a breaking change for anyone who needs DNS_CLUSTER_IP to be <IP_ADDRESS> unless they modify their code to add the necessary --dns-cluster-ip argument.

Perhaps a better approach would be to keep the existing logic, but allow the argument to override it. It's a bit messy, but avoids the breaking change.

Perhaps a better approach would be to keep the existing logic, but allow the argument to override it. It's a bit messy, but avoids the breaking change.

@micahhausler Any thoughts?

Hi @micahhausler @mogren, I have updated my PR to fall back to the existing logic if the --dns-cluster-ip positional argument is absent.
I would appreciate your review so this can be merged as it will eliminate the need for us to build this AMI in our pipeline and we want to upgrade to the newer versions of EKS.


Improve read_repl_hist() and fix running fzf

Further improvements:

authomaticaly search repl_history.jl via REPL.find_hist_file()
use '\0' as the delimiter to preserve '\n' for multiline commands in history. fzf for that use option --read0.
switching to REPL modes in which commands were executed (via https://medium.com/@Jernfrost/exploring-julia-repl-internals-6b19667a7a62 guide)

Now it works as it should!
Updated config for ~/.julia/config/startup.jl
import REPL
import REPL.LineEdit
import JLFzf
const mykeys = Dict{Any,Any}(
    # primary history search: most recent first
    "^R" => function (mistate, o, c)
        line = JLFzf.inter_fzf(JLFzf.read_repl_hist(), "--read0", "--tiebreak=index");
        JLFzf.insert_history_to_repl(mistate, line)
    end,
    # alternative history search: most relevant first
    "^T" => function (mistate, o, c)
        line = JLFzf.inter_fzf(JLFzf.read_repl_hist(), "--read0");
        JLFzf.insert_history_to_repl(mistate, line)
    end,
)
function customize_keys(repl)
    repl.interface = REPL.setup_interface(repl; extra_repl_keymap = mykeys)
end
atreplinit(customize_keys)

Alternative search <Ctrl-T> still doesn't work because (as I think) it is not predefined for REPL's search. <Ctrl-F> is used/reserved for move to right. This issue requires further investigation, but that will be another issue.

looks great, thanks. I really love the "auto switch to the correct mode"!


feat: Migrate Ethereum : eth_estimate_gas from "beerus_rest_api" to "beerus_rpc"

Feature Request
Describe the Feature Request
We are deprecating "beerus_rest_api" and now migrating all the api endpoints to "beerus_rpc".
Migrate eth_estimate_gas and add tests on beerus_rpc.
Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
(Help can be provided if you need assistance submitting a PR)

[ ] Yes
[ ] No

Self-assigned


[QUESTION] I'm having problems to save(persist) a file in disc with UploadFile

I need to save a file to disk, but so far I have only been able to clone the file, but without content on it (invalid).
Can anyone help me?

Please follow the template to ask the question. It at least asks you to provide the code you are using.

Thanks for the help here @phy25! :+1:
And thanks @gkramer01 for closing the issue. If you still have problems, you can create a new one following the template. :nerd_face:


Add new hint "Newtype field name prefixed"

New hint, turned off by default, that checks if field name in newtype constructor is prefixed with un followed by the newtype Name.
Example below:
➜  bat Foo.hs
───────┬───────────────────────────────────────────────────────────────────
       │ File: Foo.hs
───────┼───────────────────────────────────────────────────────────────────
   1   │ module Foo where
   2   │
   3   │
   4   │ newtype Foo = Foo { bar :: Bar }
───────┴───────────────────────────────────────────────────────────────────
➜  hlint Foo.hs
No hints
➜  touch .hlint.yaml
➜  echo '- warn: {name: Newtype field name prefixed}' > .hlint.yaml
➜  bat .hlint.yaml
───────┬───────────────────────────────────────────────────────────────────
       │ File: .hlint.yaml
───────┼───────────────────────────────────────────────────────────────────
   1   │ - warn: {name: Newtype field name prefixed}
───────┴───────────────────────────────────────────────────────────────────
➜  hlint Foo.hs
Foo.hs:4:1-32: Warning: Newtype field name prefixed
Found:
  newtype Foo = Foo {bar :: Bar}
Perhaps:
  newtype Foo = Foo {unFoo :: Bar}
Note: Rename field to unFoo

1 hint

Can you make this take a configurable list of acceptable prefixes, instead of hardcoding just un? In particular, a lot of idiomatic Haskell uses get and run as well as un.

@josephcsible Sure, I was hoping to extend it with an additional set of features (internally we use un for all newtypes except for wrapping monad transformers where we use run).
But I was unaware you can parameterize hints. Could you show me an example of how you can provide value to a hint (I presume you do that in .hlint.yaml)

As I actually think about how to do that, it looks like support for parameters like that isn't obvious. I'll defer to @ndmitchell for the answer to that.

@ndmitchell All checks have passed :)

+1 for "configurability" - e.g. base predominantly uses get* (getSum/getProduct etc)

+1 for "configurability" - e.g. base predominantly uses get* (getSum/getProduct etc)

@googleson78 Sure, this can get in as a separate contribution, right?

Please reopen a fresh or discuss on the ticket if people still are interested


apt-get complains about a weak digest on Debian

Hello @ll,
I get the following warning since weeks, when I run "apt-get update":

W: http://download.jitsi.org/deb/unstable/Release.gpg: Signature by key 040F57608F84BAF1BF844A62C697D823EB0AB654 uses weak digest algorithm (SHA1)

There is a Debian Wiki entry (http://wiki.debian.org/Teams/Apt/Sha1Removal) as well as https://juliank.wordpress.com/2016/03/15/clarifications-and-updates-on-apt-sha1, where it is explained in detail. It would be nice if you could change jitsi's key to SHA256 as well, so users do not get this warning ... ;)
Sorry for the inconvenience and have a nice day
Thomas.

Duplicate of #203.


Update hook_id to int64

The hook_id field  of the Audit Log API is not a string. It is represented as a number, updated the HookID to int64 to match other fields of type number.
Signed-off-by: Brett Logan<EMAIL_ADDRESS>

Working on the CLA

Pagination is also broken for the audit-log API, I've opened an issue internally as the current Pagination mechanism for this endpoint is completely different then all other endpoints. So while this change is good, it wont fix the actual audit log API. I'll open an issue here to track that problem as well.

Thank you, @lindluni !
This is a small enough change that if you need this ASAP, we could close this PR and I could open a new one (until you get your CLA working).  It's your call.


Updated Access Modal to not go back to the access list

closes #995

@jSylvestre what bug are you getting?

@jSylvestre what bug are you getting?

@wongband It is happening here:
https://github.com/ucdavis/Peaks/blob/master/Keas.Mvc/Controllers/Api/AccessController.cs#L159


Fix date formats

Last login date shows people logging in very early in the morning, and no one logging in after midday -- seems like it's not formatting dates correctly.

Duplicate of https://github.com/gchq/stroom-auth/issues/125

The Users screen is showing dates with an offset to UTC,  can we format to UTC like the rest of the application please


realm cloud notification only fired once, not during the synchronization

Goals
Show the progress when syncing the realm local db from the realm cloud
Expected Results

The syncsession progress notification will keep reporting the data during synchronization
Actual Results

The progress notification will only fire after the synchronization complete
Steps to Reproduce

Just add the progress notification from Sync Session class
Code Sample

override func viewDidLoad() {
 let session = SyncUser.current!.session(for: Constants.REALM_URL)!
 var token: SyncSession.ProgressNotificationToken?
 token = session.addProgressNotification(for: .download,
                                            mode: .forCurrentlyOutstandingWork) { progress in
                                                self.updateProgressBar(fraction: progress.fractionTransferred)
                                                if progress.isTransferComplete {
                                                    //                                                        hideProgressBar()
                                                    token?.invalidate()
                                                }
                                                else {
                                                    print("TOTAL DOWNLOAD DATA: \(progress.transferrableBytes)")
                                                }
    }
 }

private func updateProgressBar(fraction: Double) {
    print("DOWNLOAD PROGRESS : \(fraction)")
}

Version of Realm and Tooling

Realm framework version: 3.7.4
Realm Object Server version: NEWEST
Xcode version: Xcode 10 beta 4
iOS/OSX version: Mojave beta 4
Dependency manager + version: Cocoapods

@avjiang
Could you share which sync type you're using? This may be relevant.
Thanks for the information!


How to make left-option key as meta key for emacs in osx yosemite

If you are using standalone emacs, use mac-option-modifier.
(setq mac-option-modifier 'meta)

You can also do this from the menus, but what I want is the right option to still select the alternate character group, but the left option to be meta (similar to the Sun Type 4 Unix keyboard layout).  Inside emacs, I can set the right option key to be Meta, but not just the left one.  In Karabina, the I see an "Option L to Option R" setting, but no corresponding reverse option.

You can swap left and right options with new setting by this private.xml:
https://github.com/tekezo/Files/blob/master/Karabiner/private.swap_options/private.xml
How to use private.xml:
https://pqrs.org/osx/karabiner/document.html#privatexml

This worked perfectly!  Thanks so much!


conda install from conda-forge requires a geos downgrade

When I want to install shapely in the environment described in https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:cdat_conda#cloning_uvcdat_to_add_specific_packages_for_lsce, conda wants me to downgrade geos!
(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge shapely
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/share/unix_files/cdat/miniconda2/envs/cdatm14:

The following NEW packages will be INSTALLED:

    shapely: 1.5.17-np111py27_2 conda-forge

The following packages will be DOWNGRADED due to dependency conflicts:

    geos:    3.6.0-0            conda-forge --> 3.5.1-1 conda-forge

Proceed ([y]/n)? n

If I don't specifiy conda-forge, it works but gets me a slightly older version of shapely... I will assume that's good enough
(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 shapely
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/share/unix_files/cdat/miniconda2/envs/cdatm14:

The following NEW packages will be INSTALLED:

    shapely: 1.5.16-py27_0

Proceed ([y]/n)? y

We're not responsible for the conda builds. If you need, you should create an issue here:
https://github.com/conda-forge/shapely-feedstock
conda-forge does have a build of GEOS 3.6.0 so I'm not sure what's triggering the downgrade.

Thanks, @snorfalorpagus. I'm going to draft an issue template based on your comment.

Thanks @sgillies and @snorfalorpagus !


Camera of dlimg shows last drawcustom

Describe the bug
When using service: open_epaper_link.dlimg the camera of the entities targeted show the precious drawcustom (or nothing) instead of the image.
To Reproduce
Steps to reproduce the behavior:

Use a "drawcustom" to epaper 1&2
Use a "dlimg" to epaper 1
View epaper 1&2
Notice they are different
View the cameras for epaper 1&2
Notice both are still the same

Expected behavior
Camera to match any data sent from HA.
Screenshots
Additional context

Just noticed the dlimg service is deprecated. Closing the issue.


Email confirmation key missing domain

Deployment Method

[ ] Installer
[X] Docker Development
[ ] Docker Production
[ ] Bare metal Development
[ ] Bare metal Production
[ ] Digital Ocean image
[ ] Other (please provide a link Steps to Reproduce

Describe the problem*
The URL in the email to confirm an address consists of /accounts/confirm-email/ followed by the key, but it isn't prefixed by the base URL.
Steps to Reproduce

Send an email confirmation.

Relevant log output
No response

@HammyHavoc did you find a solution?

Yes! :- ) Seems that for whatever reason, base_url in config.yaml wasn't respected and wasn't visible in /settings ("No value set"), so added it via the Server Settings page, and worked like a charm.

If it's useful, my base_url in config.yaml was set as base_url: 'https://inventree.mydomain.suffix'.

Ok, I will reopen this to determine why the setting in the config file was not being respected

Ok, can confirm that the "base_url" config setting is not being "respected" as one might expect. This is a real issue

Ok, can confirm that the "base_url" config setting is not being "respected" as one might expect. This is a real issue

Just wanted to say I appreciate you and what you do and give to the world in both solutions, and your own time and energy. I hope you have the greatest year yet. 🖖 You're awesome.
Looking forward to getting more involved with contributing to Inventree however I can too.

@HammyHavoc FYI - see the updates in https://github.com/inventree/InvenTree/pull/6390
The issue on your end is that it should be called site_url (and not base_url) in the settings file :)


travis: remove unnecessary database creation

Ref: #1278

@TomasVotruba as you can see, fails because of the tmpdb stuff...

Needs a rebase due to #1286

Rebased. Looks like still some tests are skipped, so this probably would not bring any improvement.

Closing in favor of #1289


self-service data access policy

public preview feature-self-service data access policy

Microsoft authors of articles need to make updates through the private repository, so the updates can be staged and validated by current validation rules. After you review staging and fix all validation issues, you can add the sign-off comment to let the PR reviewers know the updates are ready to be merged.
It's easy to access the private repo version of your articles in GitHub. Just add -pr to the URL, as follows:
 Public repo URL: https://github.com/MicrosoftDocs/{docs-repo}
 Private repo URL: https://github.com/MicrosoftDocs/{docs-repo-pr}
We have closed this pull request. Please resubmit the updates through the private repo. For more information, see the contributor guide.


⚠ Bofrost.lu has degraded performance

In e6cfdf6, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1462 ms

Resolved: Bofrost.lu performance has improved in e565522 after 6 minutes.


[7.0.1] Delete MODULE.bazel.lock

Forked from #446

Cherry-picked in https://github.com/bazelbuild/bazel/pull/1000


Adding/Editing the default file doesn't update WebView

When editing the default file's text directly, the WebView doesn't update.
Closing/Opening the Webview won't update either, but Toggling a task's completion (in the WebView) will force a refresh.
Repro:

Open WebView
Open defualt file (text)
Add a task, save
Expected: WebView updates with new task, Actual: Nothing happens
Close/Open WebView - doesn't update either
Toggle completion (of any task) in WebView - it updates that task as well as adding your new task

I assume this is the same problem as #30.


test: existing recipes in SDKs

[x] 1. Content Management SDK: Test existing recipes
[ ] 2. Project Management SDK: Test existing recipes (wip)
[ ] 3. Conversation SDK: Test existing recipes
[ ] 4. Classroom SDK: Test existing recipes

@kpalania for Project Management SDK:

I am getting an error, while creating project in a key. Hence 4, 6, 7, 8 & 9 recipes fails.
I have copied config code and created PR


index: collect: ignore loading errors in collected index

Once we've collected stuff from original indexes, we don't really care about index loading errors and can safely ignore them.

Codecov Report
Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (9039534) 63.58% compared to head (05c987a) 63.58%.

Files
Patch %
Lines

src/dvc_data/index/collect.py
66.66%
1 Missing :warning:

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #495   +/-   ##
=======================================
  Coverage   63.58%   63.58%
=======================================
  Files          61       61
  Lines        4193     4196    +3
  Branches      715      715
=======================================
+ Hits         2666     2668    +2
- Misses       1381     1382    +1
  Partials      146      146

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Proposal: kf-app CRDs

The following is a proposed CRD for representing a kf app, called kf-app throughout this document.
In theory the kf controller will use this CRD to:

Build the app
Serve the app
Update the app if properties change
Supply an event stream

The kf-push command will:

Upload source as a container image, (currently via kontext)
Apply a new kf-app object
Listen for controller events and status conditions to replicate the build/deploy output of cf push

Here is the rough draft for the CRD itself. I followed the documentation here
Note that my use of comments. This is either for brevity or for sections which still have ambiguous requirements.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kfapps.some-group
spec:
  group: some-group
  versions:
    - name: v1alpha1
      served: true
      storage: true
  scope: Namespaced
  names:
    plural: kfapps
    singular: kfapp
    kind: KfApp
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            src-image: some.com/docker/uri:latest (uploaded via context during cf push)
            # The properties here would match those of an `application` in a CF manifest)
            # see https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#optional-attributes)
status:
  conditions:
    - # src-image updates
    - # kbuild updates
    - # kserve updates

However, CF app manifest files support multiple files. Here's a couple of push use cases:

cf push multiple app manifest: client pushes 1 image, creates n kf-app CRDs
cf push multiple app manifest but selects one app: client pushes 1 image, creates 1 kf-app. Note that this is identical to case 1 if the manifest has 1 app
cf push with no manifest: client creates a pseudo-manifest (abstracted away from the user), continues as if case 1

Now I'm not sure if CF manifests supply top level configuration which is applied to all apps in a cf manifest file. I think this sort of functionality is deprecated, but can someone confirm?
Conditions (and status in general) should be convenient for the controller; the ideas I have above are just a rough idea.
the docker app lifecycle
Cloud Foundry applications can follow the buildpack app lifecycle (src) or the docker app lifecycle (src).
In both cases an application can use a CF manifest. See the docker attribute in manifest files for more info.
However, we use buildpacks.io, also known as Cloud Native Buildpakcs, which is a slightly different thing, and I do not know if buildpacks.io support docker images in the same manner CF does.  ¯\(ツ)/¯
To support both (and in general, multiple) lifecycles, we need a way to detect which one an app follows.
I believe cf does this by knowing that the docker and buildpacks attributes must be mutually exclusive. We could do the same check when creating the schema, or even better put this logic into the validation schema.
Alternatively, this "detection" could happen client side, and the CRD could have a lifecycle_type string field. I would recommend against this in the initial design, because it seems we could add it later with little consequence.

Couple of thoughts:

Would you expound on the structure for different cases e.g. what does a docker only deploy look like compared to a source deploy?
What do you think about following Knative's lead and using the standard Kubernetes pod spec as a base? That would make our syntax similar to Pods/Deployments/KSvc/RepliaSets.
What do you think about using this CRD to manage an app's state? (started | stopped)

Would you expound on the structure for different cases e.g. what does a docker only deploy look like compared to a source deploy?

I added a section that speaks to the docker app lifecycle. It seems that CF manifest files are composed with restrictions such that CF can tell which lifecycle to take by the manifest structure. From the cf docs:
Note: Using the docker attribute in conjunction with the buildpacks or path attributes will result in an error.

In the docker case, src-image would be the compiled image, and no build pipeline would be required.

What do you think about following Knative's lead and using the standard Kubernetes pod spec as a base? That would make our syntax similar to Pods/Deployments/KSvc/RepliaSets.

I envision this CRD having top-level knative, istio, and tekton resource templates under it, but not pods.
In the link you posted: PodSpec is referenced in a RevisionSpec, and a RevisionSpec is referenced in a Revision, but a PodSpec is not referenced on a Revision or any higher level concept. So since kf-app is the root level abstract, why should it concern itself with the actual pods?

What do you think about using this CRD to manage an app's state? (started | stopped)

Yes, thats my intention. The CRD's status would contain the top level status string, plus the status of the underlying components. The controller would be what sets this information.

I envision this CRD having top-level knative, istio, and tekton resource templates under it, but not pods. In the link you posted: PodSpec is referenced in a RevisionSpec, and a RevisionSpec is referenced in a Revision, but a PodSpec is not referenced on a Revision or any higher level concept. So since kf-app is the root level abstract, why should it concern itself with the actual pods?

Here's the deck where they talk about inlining PodSpecs as the base for Knative 0.6 and beyond: https://docs.google.com/presentation/d/10wuLMFXyol731WKuO5x7lalWrH0A6YVHa4exIERQaQ8/edit Their argument is that Deployment, ReplicaSet, DaemonSet, StatefulSet, and Jobs all embed PodSpecs so they should too rather than inventing their own template.
My general thought was that we should probably reflect the look and feel of Knative/K8s rather than copying fields directly over from the app manifest. For example, cf app manifests have an env map, but maps are an anti-pattern in K8s. It sounds like you're planning on doing a template like thing to solve that so I'm happy.

Yes, that's my intention. The CRD's status would contain the top level status string, plus the status of the underlying components. The controller would be what sets this information.

Ah, I was thinking we should be able to set it on the spec field as well so a user could stop an app by setting the field to "stopped" or the like. I filed an issue with Knative serving and it looks like a feature we'll have to implement at this level rather than farther down: https://github.com/knative/serving/issues/4262

@mattysweeps and I had a whiteboarding session to fill in the details for the CRD.
Specifically, we wanted to ensure it could support the following operations:

push
start
stop
scale
restage
stage
restart

Under the following scenarios:

An operator patches a base image, and wants to rebuild apps but doesn't have the source.
A developer wants to change some property of the app without needing to rebuild.
A developer sees that something went wrong during the build and wants to re-trigger without changing anything.
A developer or operator wants to take an applicaiton down temporarially.

apiVersion: kf.dev/v1beta1
kind: Application
metadata:
  name: some-app
spec:
  source:
    uid: # if changed, forces a rebuild, even if all other fields are the same.
    docker:
      image: gcr.io/my-company/myapp # the path to the docker image
      credentials: # a secret containing the credentials to connect to the image repo
    buildpackBuild:
      source:
      stack:
      buildpack:
    # potentially extended in the future for other types of build
    # like copying a docker image from an external registry to a
    # local one, builds from Dockerfiles, etc.
  template:
    uid: # if changed, forces a restart, even if all other fields are the same.
    scale:
      min: 0 # minimum scale
      max: 0 # maximum scale, 0 for inf
    spec:
      # This is a PodSpec minus the image name.
      # It holds environment variables, quotas, etc.
      # It will be copied directly to a Kservice once the build
      # completes.
  routes:
    # This is an abstraction over a VirtualService that provides
    # cf-like functionality.
  services:
    # this field will hold references to secrets along with
    # the metadata needed to turn them into VCAP style
    # environment variables.

  # if false, the app is "stopped", serving.service gets scheduled
  # a dummy container and we tell Istio to 404 all the routes.
  # this is at the top level because it affects multiple resources.
  serve: true | false
status:
  build:
    # status of the build
    image: # the image the build says we should use for serving
  service:
    # status of the service
    # probably embedding the serving.service
  routes:
    # status of the routes
  services:
    # status of the attached services

The source, routes and services fields will be broken out into further
(hidden) CRDs that can be implemented independently and contributed back upstream.

@josephlewis42 I'm very excited about this. I had a few questions/comments:

Does the stage operation map to cf push myapp -i 0?
In regards to docker.credentials, should this be a service account instead? That seems to be the normal way knative receives this type of information.
So the implication of having stack and buildpack listed separately is that the space configuration (that will drive what builder the space uses) will need to instead have a collection of builders. The reason for this is, a builder can only reference a single stack. Therefore if we want to support the use case where CF upgraded from cflinuxfs2 to cflinuxfs3, we would have a builder for each.
Scale's sentinel values are confusing. If I set scale.min=0 and scale.max=0, I think I would expect the number of instances to stay 0. However, this looks like it would simply set it normal autoscaling. How would I scale the app to 0 (I see that I would need to set serve=false, however I'm not sure I find this intuitive)? Could we instead use -1 or just recommend a huge number?
serve seems redundant to scale in some way (This assumes the sentinel values are adjusted).

Does the stage operation map to cf push myapp -i 0?

I think that would depend on how we mapped the -i flag. If we mapped it to also ensure the app was stopped at 0, then yes.

In regards to docker.credentials, should this be a service account instead? That seems to be the normal way knative receives this type of information.

That's a good question. Today a user can set credentials in the app manifest, but we didn't want to have those plaintext in the cluster. Maybe we could implement a hook similar to Secrets that translates plaintext values to a service account or secret and replaces the values with that? WDYT?

So the implication of having stack and buildpack listed separately is that the space configuration (that will drive what builder the space uses) will need to instead have a collection of builders. The reason for this is, a builder can only reference a single stack. Therefore if we want to support the use case where CF upgraded from cflinuxfs2 to cflinuxfs3, we would have a builder for each.

We debated a bit what we wanted stack to mean in this context and weren't fully sure. WDYT about allowing that field to contain either an alias ("cflinuxfs3", "alpine") or a full buildpack builder image?
It would be the responsibility of a space to ensure that any apps put in that space had a valid image chosen from a list of available ones. The space could hold aliases for each image and replace on the fly so you could have multiple builders supporting say "alpine" or "cflinuxfs3". That would also allow us to support things like a heroku stack.

Scale's sentinel values are confusing. If I set scale.min=0 and scale.max=0, I think I would expect the number of instances to stay 0. However, this looks like it would simply set it normal autoscaling. How would I scale the app to 0 (I see that I would need to set serve=false, however I'm not sure I find this intuitive)? Could we instead use -1 or just recommend a huge number?
serve seems redundant to scale in some way (This assumes the sentinel values are adjusted).

It is confusing, but I'd like to keep the 0 value being the default for when we translate to/from go structs if possible. I think we need some other field to keep track of state so we can stop/start without losing whatever we'd set before. Maybe it would be less confusing if we brought it into the top level with something like this:
instances: # scale to n
  exactly: 3

instances: # scale to 0
  max: 5

instances: # stopped
  stopped: true
  min: 1
  max: 5

Where the implementation of instances is:
instances:
  stopped: # true if the service and routes should stop traffic
  exactly: # exactly this number can run
  min: # minimum of this number
  max: # maximum of this number, 0 = infinite

Maybe we could implement a hook similar to Secrets that translates plaintext values to a service account or secret and replaces the values with that? WDYT?

I like this idea. I think it warrants its own proposal and discussion. Would developers store their own credentials in this same way? I believe cf cups was often used for this before, but that was less than desirable.

I've started this in the feature/kfapps branch, but unfortunately I am blocked by https://github.com/kubernetes/kubernetes/issues/79149
Update: no longer blocked by this


feat: listen for the logic of requestHandlersMap config update

增加支持监听 requestHandlersMap config 更新的逻辑。

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

合并分支需要指向develop
SLA 麻烦签署下


[NIFI-5009] [NIFI-5008]: Addressing restricted component annotations

NIFI-5009:

Fixing required permission for PutParquet.

NIFI-5008:

Ensuring all restricted components are tagged as such.

Looks good, will merge


[Hub Generated] Review request for Microsoft.ApiManagement to add version stable/2019-01-01

If you are a MSFT employee you can view your work branch via this link.
Contribution checklist:

[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-java

Automation for azure-sdk-for-python

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-js

Automation for azure-sdk-for-go

Can one of the admins verify this patch?

Related to this issue - https://github.com/Azure/azure-sdk-for-js/issues/2613


Philosophical changes and future development

Thanks for your work on this project; it's come a long way from just a transliteration/functional Python fork of the C whois implementation and I use it in quite a bit of automation. I appreciate your openness to PRs and your handling of issues
What follows is not a typical issue. It's some opinion, some analysis, and an offer that I don't necessarily expect you to accept
Proposed Change
First, an obvious observation: much of the activity in Issues/PRs is "add support for xyz TLD/gTLD". You of course know that this usually requires one of the following:

A class for the TLD/gTLD
enhancements/modifications to the existing class if applicable (changes to the regex)
Addition of conditionals and/or imports (of the TLD parser class) in many cases

There are some things in the current implementation that can be done differently that provide some benefit- either technical changes, or changes to the higher level approach
I couldn't find the note I thought I had posted here a few years ago- it can summarized as "are you open to refactoring?" to which you responded (iirc) with an enthusiastic "yes". If that's no longer the case (or was imagined) then you can ignore the rest of this 😀
The local fork I have makes a foundational change to the way individual TLDs are handled in the code. Rather than requiring a large number of imports or a very large file for the classes, I used a pluggable approach; each of the TLD classes lives as a class in a module under a whois.tlds package. So, along the lines of whois.tlds.com::WhoisCom
Each class still inherits from a common one for the regex and such, but has an extra attribute:
tld: str
To identify which class to use while also avoiding hundreds if imports, I added a module that uses importlib to iterate over all of the modules at run-time, to build a global mapping of TLD -> class, by iterating over all of them and referencing the new tld attribute
I'm not sure if I dis this in the Pythonic way, but it's pretty small, only a few hundred lines or so. And it's mostly hidden behind a simple interface, along the lines of:
gets WhoisCom
cls = get_tld_class("com")
w = cls(data)
This means many of the long if/elif/elif stuff can go away. Additionally, all of the imports can go away. It's also (imo) simpler to make changes or additions to, and of course easier to read, even if only because it's less to look at
State of the fork
What I have is functional and used in daily automation, but along the way I made a few invasive changes that may make it unappealing or difficult to accept as a single merge:

Removed the non-domain lookup functionality
Added filesystem-based caching of responses (very helpful during development and when used at scale in an automated fashion)
Lots of minor stuff that touches every file (style, aesthetics, trying to adhere to PEP-8 and whatever "black" prefers)
"Breaking" changes; specifically PEP-484 type annotations which I use somewhat religiously with mypy. These annotations break Python2

For item 1, I felt it was no longer useful to support non-domain lookups, because RDAP infrastructure for other objects is in place now, and RDAP is superior to WHOIS
For item 2, what I added is relatively simple and uses cache_to_disk for the caching. It's just a decorator and slightly special exception handling which (incidentally) I contributed specifically for this; you'll notice that the README example for that project is whois-themed :)
Are you still/were you ever interested in entertaining a somewhat ambitious refactor and/or adding the caching or typing? If so, what sort of time and interest might you have to work on it if I was to give you the functional code I have?
I don't expect you to take this on; I had fully intended to see this all the way through with my own time, via a proper public fork and PRs for individual changes, but I've had a lot of time commitments since then, so it's remained a private fork. It works great for what I do but I wouldn't expect any reasonable person to accept it as a PR; it's more invasive than is necessary and as I mentioned, some features were completely removed (including Python2 support)
I'm at the point where it seems like development of the fork will die in my own private (non-Github) git repo, as it's "good enough" for me as it is now
Parsing Patterns: Per-WHOIS server rather than per-domain
This is only a thought. I didn't implement it...
I noticed while working on the fork and (particularly) while using it to look up domains at a relatively large scale (~10k domains, across many hundreds of TLDs/gTLDs) that the format of the WHOIS response doesn't necessarily correlate with the TLD. Most correctly, it correlates with the WHOIS server implementation/name. One of the more common servers for example is the Donuts WHOIS server which is the authority for tons of TLDs- either directly or through a referral. The response data is uniform across many different TLDs/gTLDs
Seeing it so many times across different TLDs made me think it might be wiser to rewrite with the objective being having server-specific parsing classes, rather than TLD-specific ones. This way you end up with much fewer parsing classes, they may end up more reliable, and they'll cleanly support parsing responses for domains that aren't explicitly supported, without requiring use of a fallback class/pattern. Realistically many of the TLD-specific parser classes might need to stick around, but it's possible that most could go away. The logic could check first for a server-specific class, then a TLD-specific one
Functionally, this may not be a huge improvement- the fallback in the current project is pretty darned good and it's relatively rare that the response isn't parsed well-enough; but it could be a good opportunity to both eliminate a lot of boilerplate/repetitive code that has really added up over and to reduce the trickle of Issues and PRs along the lines of "add support for .bla TLD", when the enhancement is typically a matter of copy/paste/rename
Finally, the end
Any thoughts? I can push what I have locally into a GitHub fork if you'd like to see exactly how much effort is involved. Or, if you're not interested and/or don't have time, I can leave it to rest. I just don't have time to do this as I intended, which originally included cleaning it up more, restoring the few features I removed, and making it merge as cleanly as possible
Thanks again and kudos on this project. 5 years ago I would have said that WHOIS was nearing its death as a result of RDAP adoption, but from what I've read, we're far from that point when it comes to TLDs. RDAP adoption for domains has completely stalked for a lot of TLDs/gTLDs, unfortunately
I'll stop taking up your time now 😀

I have not had time to finish (or even clean up) what I have, unfortunately. I do still intend to get around to it, I just can't see any time for it in the next 6 months. It does fully work for my use-case as-is (it's not a "proof of concept") but there are lots of things preventing it from being suitable for a PR
If someone is willing to work on it, it requires a bit of time. The more familiar you are with the existing codebase, the quicker the work will be, though
If anyone is interested in doing any of these things, I can put up what I have as a temporary separate repo
I'd rather not go through the trouble of doing that if nobody currently has the time or interest to take it on yet, because like everyone, I'm short on time. But if anyone speaks up I will absolutely do it and link to it here
The main work required to make this even close to suitable for a PR follows... sorry for the large dump of items

Remove (or clean up and make more tunable) the caching I added. Not sure if anyone wants caching? I added it because my use-case involves large batches done on a regular basis, and doing thousands of queries daily results in blocks occasionally. Also, it makes dev/testing faster (instant responses. @richardpenman, any thoughts on having the caching? It's applied around the outermost UDP request for a given TLD. It doesn't cache failed responses (such as network failures) but it may cache "negative" responses (I don't recall with certainty)
Restore the IP whois feature(s). I stripped out the support for doing lookups on IPs because at this point, RDAP is implemented by all of the RIRs for IP addresses and it's superior. @richardpenman I assume you want the IP functionality to remain, correct? I don't want it, but it's understandable to want to keep it intact
Review the many, many existing classes and their respective regexes and consolidate them into as few classes as possible
Analyze the classes resulting from step 3 with the WHOIS server implementation (rather than a ABC, DEF, GHI class all having the same patterns);. This is a step towards shifting from TLD-specific classes to WHOIS server implementation classes
Lots of minor and major refactors need to be cleaned up in general
Remove/replace/improve logging I added during testing. In theory it can at least be silenced easily if it isn't desired as I used the Python logging package. But maybe it should be kept, in which case it needs cleanup and an interface to control it via the CLI
Restore the CLI/entry-point. I only implemented a very simple/limited CLI interface for what I needed, which is invoking a method with argv[1] and nothing more
Documentation, specifically for developers needing to add additional parsing. Though in theory, this really ought to become an even rarer thing

Regarding the third and fourth items...

One of the consequences of the current implementation is that there are many different classes (with slightly or very different regex patterns) for the exact same output format. This happened incrementally as users contributed classes and/or patterns as one-off enhancements to an existing or new TLD. In some cases the most comprehensive/correct class for the format observed was not copy/pasted
I have not formally enumerated and documented the various WHOIS implementations encountered. It's not necessarily hard but can be time consuming. So e really dirty logging I added (logging every raw response to disk) makes it easy to collect the data but it still requires eyes/time

This could be done in stages. One way to approach it would be to transition to the importlib-based approach of getting the parser for the response as the first step
It's a big task (big change to the code) but it's mostly done. Part of that task involves splitting out each existing TLD class into its own file, in a new subpackage/directory. So you have a structure like this:
whois/
  tld/
    __init__.py
    adult.py
    ba.py
    ...
    com.py
    ...
  ...

That was a painful task but I already have it done so it's easy to copy
The other parts are the module with a few functions that scan the TLD subpackage directory to build a simple registry (a dict if tld -> class) and simple logic to choose the class from said registry at runtime, so it can be instantiated for parsing
Sorry for the empty (so far!) promises and the very long post 😞

Sorry for delay replying. Busy with babies...
Yes am very open to refactoring, particularly if it improves test coverage. I have been accepting most PRs and haven't pushed back for tests.
Instead of iterating over all the modules at runtime to build the TLD map, could importlib be used to just import the module matching the given domain TLD?
I'm not familiar with RDAP, but often people in the issues have use cases for IP lookups, so I would be hesitant about removing this. Does maintaining support for IP lookups complicate the implementation?
Caching is a great idea during development. I personally use the pdict module for this use case.
Black / PEP-8 fixes would be welcomed. Also a github CI PEP-8 check to make sure this is maintained.
I'm also open to removing Python2 support to simplify things. I've received a number of issues for Python3 but none for Python2 lately, so I think it's time to drop this.

Analyze the classes resulting from step 3 with the WHOIS server implementation (rather than a ABC, DEF, GHI class all having the same patterns);. This is a step towards shifting from TLD-specific classes to WHOIS server implementation classes

That is a great idea, though would it be harder to maintain in future when someone comes along with a missing TLD?
It would be ideal to get any of these changes in separate PRs though if you're still motivated, which given my delay in responding I don't expect!

Made some changes today to migrate codebase to pep-8 with black, removed python2 support, and added CI checks for flake8 and unit tests.

Sorry for delay replying. Busy with babies...

That's ok, we'll deduct it from all the pay you receive for your thankless dedication to this project 😉

Yes am very open to refactoring, particularly if it improves test coverage. I have been accepting most PRs and haven't pushed back for tests.

👍

Instead of iterating over all the modules at runtime to build the TLD map, could importlib be used to just import the module matching the given domain TLD?

Good question. I vaguely recall a couple of reasons I chose walking the directory and disregarding the filename:

Handling TLDs with reserved or otherwise "special" characters in them. For example, "co.uk"; I wasn't sure if importlib would be happy with importing tlds/co.uk.py. Rather than looking further into it, or using a simple "hack" like replacing the dots with underscores, I dodged the issue. There are other ways of working around this beside the simple "." -> "_" method
Handling bad input. Another example of something that can be handled by a simple solution, like sanitizing the input. However, I preferred to just dodge the issue entirely, rather than allowing external input to influence an import at all. Ultimately a quirky/arbitrary judgement call I made in retrospect

Neither of these make your approach objectively unsuitable. I don't have much of a technical argument behind what I chose, nor do I feel too strongly about it as it's a reasonably simple implementation detail to change
What is your interest in the approach you mentioned? Efficiency, from not needing to open all the files? Or reduced code? Maybe both. Any others?

I'm not familiar with RDAP

You should look into it, the gist is that RDAP is effectively WHOIS but using HTTP with consistently structured JSON responses
It's really quite nice and allows (among other things) throwing away any pattern matching logic and UDP code. And for the users, it works with HTTP proxies (this is important for my use-case)
It's really a shame it hasn't been adopted as planned for TLDs... 5 years ago I honestly expected I would no longer have a use for this project - did not work out that way at all!

but often people in the issues have use cases for IP lookups, so I would be hesitant about removing this.

Fair enough

Does maintaining support for IP lookups complicate the implementation?

I don't recall how much it complicated things- it's possible I just made a hasty judgement call.  I doubt it's a substantial hindrance

Caching is a great idea during development. > I personally use the pdict module for this use case.

👍

Black / PEP-8 fixes would be welcomed. Also a github CI PEP-8 check to make sure this is maintained

Are you familiar with pre-commit? It's very nice for helping to enforce this.. well.. pre-commit. You'll still want CI checks, but pre-commit reduces the number of reactive code changes (you may already have it in the repository, I'm on mobile and didn't check)

I'm also open to removing Python2 support to simplify things. I've received a number of issues for Python3 but none for Python2 lately, so I think it's time to drop this.

I see you already started on some of this 👍

Analyze the classes resulting from step 3 with the WHOIS server implementation (rather than a ABC, DEF, GHI class all having the same patterns);. This is a step towards shifting from TLD-specific classes to WHOIS server implementation classes
That is a great idea, though would it be harder to maintain in future when someone comes along with a missing TLD?

It could be, but doesn't necessarily have to. It would have to be kept in mind because it would be a big point against any implementation that makes this task harder

It would be ideal to get any of these changes in separate PRs though

Yeah, makes sense. Would you do it directly to master or some new branch? I guess most users aren't impacted by changed to master unless it's published as a package, so maybe it doesn't matter, but a branch may be the best way to go to avoid confusing users installing from master (a terrible practice, but I'm sure  someone is doing it...)

if you're still motivated, which given my delay in responding I don't expect!

I was the one with the longest delay, I think. I'm still motivated, but my time is still pretty limited unfortunately
I'll keep following this and hopefully can carve at least a little time out to lend a hand

More idle commentary to follow 😀
I don't think this necessarily requires a response, you can consider it food for thought, otherwise I'll consider it a note for myself. Could be you've already considered it
First, I was recently looking through my Github updates and noticed another project I had followed a long, long time ago, which is a whois parsing implementation in Ruby
https://github.com/weppos/whois-parser/tree/main/lib/whois/parsers
Because of our recent discussion, I went to look at how that project approaches it, to see if it gave me any new ideas or perspectives.
You may be interested in checking it out, or may already be familiar with it. My quick observation from a couple of quick glances- I don't know that there's anything particularly novel there. Though it seems it may leverage classes and language features more extensively. The result of that might be more elegant (subjective) but each subclass is also a lot bulkier than with your current approach. 🤷♂
That code made me think of another point, which is the lowest level detail of the parsing. The current implementation uses a single regex string. Beautiful, as it's simple by many measures. But also ugly in some cases, because occasionally it gets ridiculous constructing a single regex for the more exceptionally unruly formats and records
Have you considered support in the base class for optional field-specific regex? This would be for really exceptional cases where a single regex is either extraordinarily difficult or seems not possible with a single regex
I have a few specific examples I'm going to dig up now. I'm proficient with regex but don't consider myself an expert. Maybe you can actually come up with a regex for them and I'll learn something new
Thanks!

Regardless of my mistakes/failing memory, I had this in my notes as well as a "potentially helpful reference" with regards to parsing - I think seeing this and one other project taking the server-specific parsing route is probably what made me decide it wasn't an insane idea and might actually be worth raising :smile:
https://github.com/Whoisdoma/WhoisParser/blob/master/src/Config/whois.ini

I gisted one specific piece of the changes on my side if you're curious about some of the larger/structural modifications. Here's the entry.py file in its entirety:
https://gist.github.com/mzpqnxow/28909f8803a99b2cf4f435d3db7e86b8
Some dubious choices for sure - like making it a subclass of UserDict. But I figured better to at least show you SOMETHING so you get an idea of what it looks like
The class "registry" file (registry.py) can be found here:
https://gist.github.com/mzpqnxow/2d3dc445a2033ef893417c525b718186
After your suggestion/question "can they just be loaded based on the name of the tld?" - I realize this does look needlessly complex. I'm still not 100% sure if I encountered an issue with the approach you described or if I just chose this method because it seemed better to me at the time :smile:

What is your interest in the approach you mentioned? Efficiency, from not needing to open all the files?

Yeah was thinking of efficiency - do you have an idea how long the imports would take? Apparently this can be measured with -X importtime: https://stackoverflow.com/a/52090274/105066

Would you do it directly to master or some new branch? I guess most users aren't impacted by changed to master unless it's published as a package, so maybe it doesn't matter

Master is fine and hopefully no production users are installing from the repo!

https://github.com/weppos/whois-parser/tree/main/lib/whois/parsers

Hadn't come across this. A pity we are re-inventing the wheel in each language!

the format of the WHOIS response doesn't necessarily correlate with the TLD. Most correctly, it correlates with the WHOIS server implementation/name.

Do you have an example? Actually I didn't realize this. If this is true then the server approach makes sense, but would be a huge change.

What is your interest in the approach you mentioned? Efficiency, from not needing to open all the files?

Yeah was thinking of efficiency - do you have an idea how long the imports would take? Apparently this can be measured with -X importtime: https://stackoverflow.com/a/52090274/105066

Should have done that, but alas, I did not :disappointed:

Would you do it directly to master or some new branch? I guess most users aren't impacted by changed to master unless it's published as a package, so maybe it doesn't matter

Master is fine and hopefully no production users are installing from the repo!

:+1:

https://github.com/weppos/whois-parser/tree/main/lib/whois/parsers

Hadn't come across this. A pity we are re-inventing the wheel in each language!

Indeed, same here
Ideally someone would/would have created some YaML files or some other language-agnostic content to describe the parsing, so that we could all use it by writing a small interpreter for that description in the language of our choice - and let The Community :tm:  maintain the parsing data files. I actually started on something like that along while ago, but it quickly got ugly and I wasn't willing to make the decisions about the tradeoffs required. At some point you have a few choices to make:

Accept that a LOT of redundant/boilerplate will be required, for servers that have overlapping parsing rules
Accept that the YaML will become slightly more complex if you want to avoid that first problem - you need to effect inheritance within YaML. It can at least be done in a lightweight fashion for the most part, if only one level of inheritance required and no multiple inheritance is required - by having a "special" key/value like inherit: base, instructing the engine to another YaML file first
Defeat the purpose of the entire thing and support the use custom deserialization methods (terrible idea)

Someone with good judgement , development skills and some time to dedicate could probably still do this and balance/minimize problems 1 and 2 while avoiding 3, but it's not going to current-me, sadly

the format of the WHOIS response doesn't necessarily correlate with the TLD. Most correctly, it correlates with the WHOIS server implementation/name.

Do you have an example? Actually I didn't realize this. If this is true then the server approach makes sense, but would be a huge change.

Actually, the best example handy can be found in the aforementioned project, rather than parse .com and .net with a class specific to each, they just check the final WHOIS server (in case of referrals) and then look up the parser based on that WHOIS server name:
https://github.com/weppos/whois-parser/blob/main/lib/whois/parsers/ccwhois.verisign-grs.com.rb
Or did you mean examples of different TLDs that go back to a common WHOIS server?

Or did you mean examples of different TLDs that go back to a common WHOIS server?

Did you mean that 2 domains with the same TLD use different WHOIS servers, and so have different formats? That's what I understood, but maybe you meant something else. If not then I don't think is worth the large restructuring and the .net parser could just inherit from the .com parser.

Or did you mean examples of different TLDs that go back to a common WHOIS server?

Did you mean that 2 domains with the same TLD use different WHOIS servers, and so have different formats? That's what I understood, but maybe you meant something else. If not then I don't think is worth the large restructuring and the .net parser could just inherit from the .com parser.

That's one example, but the most common one may be with the so-called Generic TLDs (gTLDs)
I don't know if you're familiar with how the gTLDs are different from traditional TLDs- if not, here's all you need to know:
The way gTLds work from a WHOIS perspective is how all TLDs should have worked as well, which is the owner of the gTLD MUST maintain an A or CNAME record for whois.nic.. For example, the WHOIS server for the .adult domain is guaranteed to be whois.nic.adult. It was a smart mechanism and IANA appears to have strictly enforced it, probably by helping to make it easy
Anyway, what this means in practice is that for owners that either:
A. Have multiple gTLDs all going to one WHOIS server
Or..
B. Farm out the actual WHOIS service to a third party
And
C. Accomplish this via a WHOIS protocol level referral (rather than allowing DNS to make this opaque to the WHOIS service)
... have multiple TLDs that ultimately get data from the same WHOIS server, which is known to the WHOIS client, because of the referral
There are also a cases where countries own a gTLD and fall into the above configuration, where the gTLD WHOIS server has a referral to the countries standard WHOIS server
Anyway, it's still a relatively small number of the total. Less than 5% is my estimate based on some data I looked at a few weeks ago. But it wasn't comprehensive
The upside, compared to any any real or perceived complexity associated with implementating it, would be potentially handling TLD parsing perfectly, even when there's no explicit support for the TLD (or gTLD)
This is probably not an either/or decision - could be like this:

Check if the TLD has an explicit parser, if so, use that
If not, check if the final "leaf" WHOIS server has an explicit parser, if so, use that
If neither, fall back to the default parsing class (as you do today)

Regardless of whether you think the server-based parsing is desirable, it's probably a good idea to invest a little more into the fallback behavior. Maybe make it more comprehensive/robust, by trying several fallback classes/pattern-sets, returning results from the one with the best results, as measured by the most parsed fields
This approach may not be precisely this simple, and there would be a tradeoff with efficiency- for example, how many different pattern sets should be tested? But, the general idea is not crazy or infeasible, I think
Either way, I don't think any of these approaches bloat the code too much. Could be wrong about that of course
Last thing, because it just came to mind. I don't necessarily feel too strongly about this
There is currently an insane number of classes that are either exactly the same as others (but in their own file) or exactly the sane except for a tiny modification that makes the class either inferior to or superior to the one from which it was copied. Despite the data actually being the same. It's effectively just drift from the original class that got copied many times, where the original later had some enhancement
It would be great to prevent this from happening - if the data is the same for TLD x, y and z, and the parser for x is improved, they should all be improved
It's related somewhat to the per-server class idea, except it happens without referrals- think of all the TLDs that go directly to GoDaddy, MarkMonitor or Verisign, and that have the exact same underlying data format. I believe this is a LOT, and you can tell by comparing the parser classes (it's a lot simpler once you split them out into files, where you can list by size, and diff from the shell)

Also, sorry for the jumbled up mess of a comment- I wanted to get back to you and wrote that up pretty hastily

I owe you lots more data, but I want to put a few notes here, they don't require responses. I will edit this update in the future, rather than create more noise with new comments:)
I don't suggest any of these are high-priority - they're really just notes for future reference
Observations on consistency / inconsistencies in regex for a given field
In the majority of cases, the value to the left of the colon is what's dynamic - the capturing part is typically the same, with probably some exceptions. But because of all of the incremental development over time, from various parties, there's a lot of reinvention of wheels, with some of said wheels not being quite so round
Example Case: Postal Code / Zip Code
Here's a quick summary (and a tally on the left, by how many times they occur) of how postal code/zip code are being parsed across the various classes in master as of 6/2024:
      1 (?:Administrativ contact\n.*:.*\n.*\n.*\n.*\n.*\n.*\n.*: )(.+)
      1 Administrative Contact Postal Code: *(.+)
      1 (?<=Administrative Contacts:)[\s\W\w]*?postal-code-loc:(.*)
      1 AdminPostal Code\.*: *(\d+)
      1 Admin ZipCode: *(.+)
      1 adm-zipcode: *(.+)
      1 Billing Contact Postal Code: *(.+)
      1 BillingPostal Code\.*: *(\d+)
      1 (?<=Contact)[\s\S]*Postal Code:(.*)
      1 org-zipcode:*(.+)
      1 (?:Owner Contact\n.*:.*\n.*\n.*\n.*\n.*\n.*\n.*: )(.+)
      1 Owner ZipCode: *(.+)
      1 postal code: *(.+)
      1 PostalCode: *(.+)
      1 registrant_contact_postalcode:\s*([^\n\r]+)
      1 RegistrantPostal Code\.*: *(\d+)
      1 Registrant\s*(?:.*\n){4}\s*Postalcode: *(.+)
      1 (?<=Registrant:)[\s\W\w]*?postal-code-loc:(.*)
      1 Registrant Zip Code\s*: *(.+)
      1 Registrar:\s*(?:.*\n){2}\s*(\S*)\s(?:.*)
      1 (?<=Registrar)[\s\S]*?Postal Code:(.*)
      1 (?<=Registrar:)[\s\W\w]*?abuse-postal:\s+(.*)\n
      1 Sponsoring Registrar Postal Code:(.+)
      1 (?:Technical contact\n.*:.*\n.*\n.*\n.*\n.*\n.*\n.*: )(.+)
      1 Technical Contact Postal Code: *(.+)
      1 TechPostal Code\.*: *(\d+)
      1 tec-zipcode: *(.+)
      2 Billing Postal Code: (.*)
      2 Billing Postal Code: *(.+)
      2 Tech Postal Code:(.+)
      3 Admin Postal Code: (.*)
      3 Registrant Postal Code: (.*)
      3 Tech Postal Code: (.*)
      4 Admin Postal Code:(.+)
      5 Admin Postal Code: *(.+)
      5 Registrant Postal Code:(.+)
      5 Tech Postal Code: *(.+)
     12 Registrant Postal Code: *(.+)

Or, reducing that down, accepting that we'll lose a couple of the really weird ones that have multiple colons:
› grep -Ei '(postal|zip)' parser.py | cut -d '"' -f 4 | sed -e 's|?:|?_|' | cut -d ':' -f 2- | sed -e 's|.*\?(|(|g' | sort | uniq -c | sort -n
      1 (?:.*)            <-- This one may be legit, .nl uses the annoying multiline style
      1 (.*)\n           <-- This does not appear to be necessary at first glance (for .ua)
      1 ([^\n\r]+)    <-- Also probably legit, from .eu using the multi-line style
      4 (\d+)
     15 (.*)
     51 (.+)

Presumably, the intention was for all to use (.+) , except for the three I called out as weird multi-line ones
I wonder if it might make sense to have and encourage/require use of a canned value for as many of the fields as possible, even when they're as simple as (.+). This would be consistent with how you currently provide EMAIL_REGEX. It makes it easier for contributors to do it right than to do it "wrong". And it makes everything behave the same. As it stands, non-US style postal codes (which may contain letters) and even the long-form US zip codes (\d+-\d+) will fail to parse for those few outliers using \d+
...

Here's a quick stab at a proposed change for the classes
There are some small details that change, it's still largely the scheme you implemented
This is separate from the idea about having a lookup to choose the parser class
What I like most about making the classes look more like this is that it lets new TLD classes skip having to implement any methods - even __init__(). You only need to define the attributes, put a docstring (so the interpreter can parse it) and you're good to go
Here's roughly what I had in mind:
from collections import UserDict
from typing import Any, Dict

import re

class PywhoisError(Exception):
    def __init__(self, message, text: str = None, cls: Any = None):
        super().__init__(message)
        self.text = text
        # Could receive the class, if you want to pull more info out I suppose
        self.cls = cls

class WhoisBase(UserDict):
    domain: str
    text: str
    not_found: str
    regex: Dict[str, str]

    def __init__(self, domain: str, text: str):
        UserDict.__init__(self)
        self.domain = domain
        self.text = text
        if re.search(self.not_found, text):
            raise PywhoisError("Not found !!", text=text)
        self._parse()

    def _parse(self) -> None:
        """Common parser function"""
        # The parsing logic
        # ...
        # self['registrar'] = whatever

class TLDBase(WhoisBase):
    """Not to be instantiated directly; extend for each TLD"""

    # Only needed for later, to implement lookup via the tld field if decided to do that
    tld: str

    # Parser defaults
    not_found = r"Default not found pattern"
    regex = {
        "domain_name": r"Domain name: *(.+)",
        # ...
    }

    def __init__(self, domain: str, text: str):
        self.domain = domain
        self.text = text
        WhoisBase.__init__(self, domain, text)

# A TLD class can be a little simpler/briefer now
# No code should be needed for most cases, in theory
class WhoisCom(TLDBase):
    tld = "com"
    regex = {
        "field1": "patt1",
        "field2": "patt2",
        # ...
    }
    # not_found = r"Not found"
    pattern = {
        # Same as before, still can optionally override regex ...
        "domain_name": r"Some special pattern that overrides the common one",
        # ...
    }

Some caveats ...
This might require some minor changes elsewhere
There is probably a better way to do what I'm trying to accomplish, from a simplicity perspective or flexibility perspective
I chose what is probably a hacky approach, to avoid some of the issues with multiple inheritance (since the base class is inheriting from UserDict
I'm not a classically trained CS person, and I only play a Python developer part-time, so pardon and glaring stupidity :)

@richardpenman I spent a few hours this morning on a pretty large refactor (or, it felt large - I guess maybe not)
The main focus was simplifying the sub-classes - removing the need for __init__ in each one
The summary of what I did is at the top of the README here: https://github.com/mzpqnxow/whois_/tree/boilerplate-reduction-refactor
Not sending a PR, the README will explain why - while it does "work", it's not complete - I made things worse in some places, especially depending on your taste and preference. That's OK, it's a work in progress
Happy to collaborate in whatever way works for you, on whatever timetable
I actually did something! :>


add a proper mapping gui

let user assign arbitrary binds for each button/analog input; it could be something as simple as having a dropdown for each GC input with corresponding N64 input (probably easiest to implement), or it could be n-rage style, where you click on a n64 button and press desired input on the GameCube controller.
a nice bonus would be if you could set up separate mappings for each controller, as well as automatically import configuration from old versions of the plugin

hi, i've been busy with life and other projects lately, i'll see if i can hack up something usable this week or so :o)

just made a new release with the mapping ui and a couple other stuffs :o) enjoy https://github.com/wermipls/pj64-wiiu-gcn/releases/tag/v1.0.0


Add AAY-Chart to samples

I wanted to add the AAY-Chart Library (https://github.com/TheChance101/AAY-chart) to the Kotlin Multiplatform samples list. This library offers a wide range of charts for Kotlin Multiplatform applications using Compose Multiplatform. It supports iOS, Android, desktop, and web platforms, making it versatile for visualizing data across different environments.

@zamulla Any updates!?

Hi, thanks a lot for showcasing Kotlin Multiplatform! And sorry for taking a long time to get back to you.
We are currently considering the way the Samples page should look to be most useful to our users. In the meantime, we stopped accepting PRs to this page.
We recommend everyone with a good sample of KMP in action to use the kotlin-multiplatform-sample topic on GitHub (or the kotlin-multiplatform topic for KMP projects in general).
Thanks again for reaching out!


Please add the support for Mi Fan (zhimi.fan.v3)

This is a first version Mi Fan, please add it if possible
Model : zhimi.fan.v3

Will add support for the fan in the next update!

The device should now be supported as of version 0.9.18


Compiling ARM on x64

Hi all,
It is a few weeks I am trying to set up an environment for compiling a Linux-ARM executable on a x64 machine, without much luck for now.
I have set up multiple machines including Debian 9, Ubuntu 16, Ubuntu 18, all 64-bits, even tried with 32-bits versions.  I am currently working on a Ubuntu 16 x64.  Probably an obvious question, but could someone confirm this is supported for compiling an ARM32 output?
There are a few possible typos in the following document that are possibly confusing:
https://github.com/dotnet/corert/blob/master/Documentation/cross-building.md

"In addition, to cross compile CoreCLR" should it be CoreRT?
"Cross compiling CoreCLR" should it be CoreRT?
"$ ROOTFS_DIR=~/coreclr-cross/" should it be corert?
"$ ROOTFS_DIR=~/coreclr-cross/arm ./build.sh" should it be corert?

Another aspect that I didn't get right is the section "Using CoreRT for cross compiling under arm on x86 host".  Is this the section applying to my case where I whish building an output for ARM32 on x86?  If yes, then when would the previous section "Cross compiling CoreCLR" apply?
A few more remarks on the section "Using CoreRT for cross compiling under arm on x86 host":

The steps for "Build CoreFX" seems outdated.  Should probably by updated with https://github.com/dotnet/corefx/blob/master/Documentation/building/cross-building.md.
"./build.sh x86 debug verbose cross crosstarget skiptests" There is no parameter crosstarget in build.sh as far as I can see.  Is this outdated?  I am running into various issues when trying to run this step without "crosstarget" parameter, like file not found of ConstrainedCallSupportHelpers.S, "GetUnwindProcInfo is not supported on this arch yet.", etc.
Regarding step 4, it would be helpful to give examples of the variables.  What is supposed to be ${WORKING_DIR}?  The output of the application that should be compiled?

Thank you,
Mario

@BredPet Do you still maintain cross-compiled build of CoreRT?

@jkotas Thank you for your reply.  A ready-made build would certainly be helpful to get me started and focus on my main task.
I would be glad though to continue the trials of building by myself and then contribute to updating the documentation based on my experience.  I am totally stuck at this point though so any feedback will be appreciated.

Few month ago documentation was up to date. But there were some changes in build script  options some recently. So now may be some changes are required.
The main issues with building on x64 is that CoreCLR doesn't support cross bit mode (and CoreRT is needed CoreCLR since use RuyJIT compiler from it). And for building you should make x86 rootfs, copy x86 CoreCLR to it and cross-compile inside rootfs to ARM target.

@alpencolt Thank you very much for your comments.  I am trying to create a document describing step-by-step the installation of a VM from scratch.  I am hitting some roadblocks as described in my first message.  Would you or others be interested in reviewing my notes?  I would then be more than happy to contribute to improving and updating the documentation once I have a valid procedure.

@mariopasquali I think you can create pull request with changes to documentation and mark it with [WIP]. And we will review it.
cc @kbaladurin

@alpencolt Thank you for your comment.  I will be glad doing a PR with updated documentation, but at this point I am stuck with issues (described in my first post) I can't solve due to limited expertise on the matter.  Would someone be willing to review my current notes in private to provide some guidance and hopefully get me one step further?

hi all, maybe it would be more efficient if I share the steps I did and the specific issues I have.
I tried the following steps with a Debian x64 9, Ubuntu x64 18.01 and Ubuntu x64 16.01.  I am now using mainly Ubuntu 16.01 for my latest tests.  My procedure is below.  All goes well until the last step sudo ./build.sh x86 release verbose cross arm skiptests which is the last command of step 3 of "Using CoreRT for cross compiling under arm on x86 host" in https://github.com/dotnet/corert/blob/master/Documentation/cross-building.md.
I am getting a file not found of /corert/src/Native/System.Private.TypeLoader.Native/i386/ConstrainedCallSupportHelpers.S.  The build go on though and I also see the error bits/c++config.h file not found.  Any hint on how to solve this?
# Install CoreRT Prerequisites
# https://github.com/dotnet/corert/blob/master/Documentation/prerequisites-for-building.md
echo "deb http://llvm.org/apt/xenial/ llvm-toolchain-xenial-3.9 main" | sudo tee /etc/apt/sources.list.d/llvm.list
wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | sudo apt-key add -
sudo apt-get update

sudo apt-get install cmake clang-3.9 libicu-dev uuid-dev libcurl4-openssl-dev zlib1g-dev libkrb5-dev

# Install CoreRT Prerequisites for cross-building
# From https://github.com/dotnet/corert/blob/master/Documentation/cross-building.md
sudo apt-get install qemu qemu-user-static binfmt-support debootstrap
sudo apt-get install binutils-arm-linux-gnueabihf

# Add 32-bit multiarch-support on x64 host
echo "foreign-architecture i386" > /etc/dpkg/dpkg.cfg.d/multiarch
sudo apt-get update
sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386
sudo apt-get install multiarch-support
sudo apt-get install gcc-multilib

# Install dotnet SDK
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
sudo mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/9/prod.list
sudo mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
sudo chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
sudo chown root:root /etc/apt/sources.list.d/microsoft-prod.list

curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2

# Install git
sudo apt install git

# Clone dotnet projects
mkdir dotnet
cd dotnet

git clone https://github.com/dotnet/corert.git
git clone https://github.com/dotnet/corefx.git
git clone https://github.com/dotnet/coreclr.git

## https://github.com/dotnet/corert/blob/master/Documentation/cross-building.md

# Cross-Build CoreCRL
# https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md
cd coreclr
sudo apt install cmake
sudo apt install clang

sudo ./cross/build-rootfs.sh arm
sudo ./build.sh arm release verbose cross
cd ..

# Cross-Build CoreFX
# https://github.com/dotnet/corefx/blob/master/Documentation/building/cross-building.md
cd corefx
sudo ./cross/build-rootfs.sh arm
sudo ./src/Native/build-native.sh release arm verbose cross
sudo ./build.sh /p:BuildNative=false
cd ..

# Cross-Build CoreRT
# https://github.com/dotnet/corert/blob/master/Documentation/cross-building.md

cd corert
sudo ./cross/build-rootfs.sh arm
sudo ./cross/build-rootfs.sh x86 xenial
sudo ./build.sh arm release verbose cross skiptests
sudo ./build.sh x86 release verbose cross arm skiptests

I am getting a file not found of /corert/src/Native/System.Private.TypeLoader.Native/i386/ConstrainedCallSupportHelpers.S

Can you try creating this file? You can give it the same contents as e.g. https://github.com/dotnet/corert/blob/f158ae90111d84f0e5f1500f485ec9ce914e523d/src/Native/Runtime/i386/AllocFast.S

Thanks.  I did as suggested for the following 3 files.  Strange thing is that these already exist in the folder with .asm extension.  Maybe the build script is not picking the correct files?
ConstrainedCallSupportHelpers.S
MethodEntrypointStubs.S
VTableResolver.S

With this now the build went further, and I am hitting the following errors.  Any idea on what could be wrong?
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:34:2: error: "Unwinding is not implemented for this architecture yet."
#error "Unwinding is not implemented for this architecture yet."
 ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:443:6: error: "Unwinding is not implemented for this architecture yet."
    #error "Unwinding is not implemented for this architecture yet."
     ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:447:19: error: use of undeclared identifier 'uc'
    bool retVal = uc.getInfoFromDwarfSection(pc, uwInfoSections, 0 /* fdeSectionOffsetHint */);
                  ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:454:5: error: use of undeclared identifier 'uc'
    uc.getInfo(&procInfo);
    ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:456:42: error: use of undeclared identifier 'Registers_REGDISPLAY'
    DwarfInstructions<LocalAddressSpace, Registers_REGDISPLAY> dwarfInst;
                                         ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:458:86: error: use of undeclared identifier 'Registers_REGDISPLAY'
    int stepRet = dwarfInst.stepWithDwarf(_addressSpace, pc, procInfo.unwind_info, *(Registers_REGDISPLAY*)regs);
                                                                                ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:458:107: error: expected expression
    int stepRet = dwarfInst.stepWithDwarf(_addressSpace, pc, procInfo.unwind_info, *(Registers_REGDISPLAY*)regs);
                                                                                ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:458:19: error: use of undeclared identifier 'dwarfInst'
    int stepRet = dwarfInst.stepWithDwarf(_addressSpace, pc, procInfo.unwind_info, *(Registers_REGDISPLAY*)regs);
                  ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:500:5: error: use of undeclared identifier 'dl_iterate_cb_data'; did you mean 'dl_iterate_phdr'?
    dl_iterate_cb_data cb_data = {&uwInfoSections, pc };
    ^~~~~~~~~~~~~~~~~~
    dl_iterate_phdr
/usr/include/link.h:167:12: note: 'dl_iterate_phdr' declared here
extern int dl_iterate_phdr (int (*__callback) (struct dl_phdr_info *,
           ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:500:23: error: expected ';' after expression
    dl_iterate_cb_data cb_data = {&uwInfoSections, pc };
                      ^
                      ;
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:500:24: error: use of undeclared identifier 'cb_data'
    dl_iterate_cb_data cb_data = {&uwInfoSections, pc };
                       ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:501:21: error: use of undeclared identifier 'LocateSectionsCallback'
    dl_iterate_phdr(LocateSectionsCallback, &cb_data);
                    ^
/dotnet/corert/src/Native/Runtime/unix/UnwindHelpers.cpp:501:46: error: use of undeclared identifier 'cb_data'
    dl_iterate_phdr(LocateSectionsCallback, &cb_data);
                                             ^

Hi, any idea if this idea could be because I didn't configure something correctly, or if it is rather an issue in the build scripts?
Anyone having success doing this cross-compile build who could shed some light to help me progress?
Thanks!

Hi again...  I realize you might all be busy.  No one is having this issue, or am I the only one trying this configuration?  Any hint if I set up something wrong, or if there is something wrong on the current code base?  @jkotas @alpencolt @MichalStrehovsky  Anyone having some sympathy?  :)

You are likely the only one trying this configuration in a while. This configuration is not used by many people. There are likely issues in the current master that crept in over time.
You can try disabling the runtime x86 projects with build breaks to see whether it will get you further, e.g.: https://github.com/jkotas/corert/commit/db084d25731df61b927510b0201c1d813b08e17f . These projects should not be needed to get a working arm cross-compiled environment.


Fix ingress template for GKE comptibility

I had problems analog to the issue linked below and the same solution worked for me aswell. I hope this does not conflict with any other implementation.
https://github.com/jetstack/kube-lego/issues/115#issuecomment-290235071

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi thanks for the PR, I have a question, does the cli logs stream correctly?

Just read the issue you linked, and it should work, not sure if this will break nginx though.

Hi thanks for the PR, I have a question, does the cli logs stream correctly?

I am using the same config on NGINX and polyaxon experiment -xp 4930 logs or polyaxon experiment -xp 4930 logs --follow doesn't work. Only polyaxon experiment -xp 4930 logs --past works. Though I am not sure if that's related.

Thx for checking that @lgeiger, I didn't find the time yet.

@lgeiger that's exactly what I wanted to know.
Seems like this is only for GKE ingress, why do you need that for vanilla nginx ingress? I am sure I tested the current config with an nginx ingress from stable helm chart and it works.

Seems like this is only for GKE ingress, why do you need that for vanilla nginx ingress? I am sure I tested the current config with an nginx ingress from stable helm chart and it works.

When I setup the GKE ingress, Polyaxon didn't load the CSS assets. This seamed to fix it.
I can try to remove this "fix" next week and we can see if it still works (We have a paper deadline coming up so I won't play with our "production" deployment right now).

I removed the fix by editing the ingress deployment and the polyaxon UI seams fine 🎉
Unfortunately polyaxon experiment logs still fails with the following error:
Setting OrderedDict([('check_count', 12), ('current_version', '0.4.4'), ('min_version', '0.4.2'), ('log_handler', OrderedDict([('dsn', 'aHR0cHM6Ly84MTc4MjEwYjY4NDU0OGE1Yjk2YjgxMzFmYTY0NWM5ZTozY2IwZGI1OTY5NDU0NjNkYTUzMmUzNmRiZTQ4NjFmNUBzZW50cnkuaW8vMTE5NzQxNg=='), ('environment', 'production'), ('tags', {'cli_min_version': '0.4.2', 'cli_latest_version': '0.4.4', 'platform_min_version': '0.4.2', 'platform_latest_version': '0.4.4', 'chart_version': '0.4.4'})]))]) in the file .polyaxoncli
Starting request to url: https://polyaxon.mydomain.com:443/api/v1/versions/cli/ with params: None, data: None
Starting new HTTPS connection (1): polyaxon.mydomain.com:443
https://polyaxon.mydomain.com:443 "GET /api/v1/versions/cli/ HTTP/1.1" 200 48
Response Content: b'{"latest_version":"0.4.4","min_version":"0.4.2"}', Headers: {'Server': 'nginx/1.10.3', 'Date': 'Thu, 30 May 2019 20:13:53 GMT', 'Content-Type': 'application/json', 'Content-Length': '48', 'Allow': 'GET, HEAD, OPTIONS', 'X-Frame-Options': 'SAMEORIGIN', 'Vary': 'Origin, Cookie', 'Via': '1.1 google', 'Alt-Svc': 'clear'}
Setting OrderedDict([('check_count', 12), ('current_version', '0.4.4'), ('min_version', '0.4.2'), ('log_handler', OrderedDict([('dsn', 'aHR0cHM6Ly84MTc4MjEwYjY4NDU0OGE1Yjk2YjgxMzFmYTY0NWM5ZTozY2IwZGI1OTY5NDU0NjNkYTUzMmUzNmRiZTQ4NjFmNUBzZW50cnkuaW8vMTE5NzQxNg=='), ('environment', 'production'), ('tags', {'cli_min_version': '0.4.2', 'cli_latest_version': '0.4.4', 'platform_min_version': '0.4.2', 'platform_latest_version': '0.4.4', 'chart_version': '0.4.4'})]))]) in the file .polyaxoncli
Termination cause: Handshake status 404 Not Found
Session disconnected.
Session ended

Should I open a new issue for this?

next release there will be one port exposed, so it should be fixed in that case.

In the meanwhile I could reproduce the issue (logs don't get streamed). Will try nginx ingress next. Anyway I think we can close this.

Sadly, after changing to nginx ingress, log streaming is still not working for me. But I don't need to edit the path and the webinterface looks ok. :tada:
I'm not sure if this could be related but I get SSL verification warnings (see below) with every API call (because verification is off. If I enable it I get verification errors. feel free to check: https://polyaxon.magazino.eu )
@mouradmourafiq If this is not related I would open two separate issues. (here in the chart repo I guess?)
Example of SSL warnings and failing log stream
$ polyaxon run -u -l --ttl 120
/home/muenst/polyaxon_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
[================================] 8145/8145 - 00:00:00
Files uploaded.
Creating an independent experiment.
/home/muenst/polyaxon_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Experiment `66` was created
Session ended

This is the generated ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  creationTimestamp: 2019-06-17T13:25:11Z
  generation: 1
  labels:
    app.kubernetes.io/component: polyaxon-api
    app.kubernetes.io/instance: polyaxon-0.4.4
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/name: polyaxon-polyaxon-ingress
    app.kubernetes.io/part-of: polyaxon-core
    app.kubernetes.io/version: 0.4.4
    chart: polyaxon-0.4.4
    heritage: Tiller
    release: polyaxon
    role: polyaxon-api
    type: polyaxon-core
  name: polyaxon-polyaxon-ingress
  namespace: polyaxon
  resourceVersion: "12437542"
  selfLink: /apis/extensions/v1beta1/namespaces/polyaxon/ingresses/polyaxon-polyaxon-ingress
  uid: 558c4f2c-9103-11e9-9b76-42010aa40008
spec:
  rules:
  - host: polyaxon.magazino.eu
    http:
      paths:
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 80
        path: /
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 1337
        path: /ws
  tls:
  - hosts:
    - polyaxon.magazino.eu
    secretName: polyaxon-cert
status:
  loadBalancer:
    ingress:
    - ip: <IP_ADDRESS>

I think the easiest workaround here to allow people to customize the path is to update the PR with:

I am closing this PR, I will push the change myself for the next release. Thank you for the explanation and for creating the PR.

Thank you! Sorry for the late response... I must have missed the notification.
Meanwhile, I was busy with other tasks while our installation of 0.4.4 was running very well and creepingly transitioned from a test installation to a loved tool used in everyday development. This week I'm going to attempt the migration to 0.5.x and I'm very much looking forward to it :)