access violation reading 0xFFFFFFFFFFFFFFFF

Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Users\Gerbuiker\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\Gerbuiker\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:/Users/Gerbuiker/PycharmProjects/LabEnvironment/receive_data.py", line 14, in read_stream
    ch_labels.append(ch.child_value("label"))
  File "C:\Users\Gerbuiker\Documents\Virtualenvs\HITLab\lib\site-packages\pylsl\pylsl.py", line 985, in child_value
    res = lib.lsl_child_value_n(self.e, str.encode(name))
OSError: exception: access violation reading 0xFFFFFFFFFFFFFFFF

Happens sporadically so I'm not sure how to reproduce it but I'm reading multiple streams, one stream per thread.

pylsl isn't great at threading. This is a known issue in Linux. I didn't think it was a problem in Windows but I guess I'm not surprised to see that it is.
An annoying workaround is to use a single thread to poll all the streams then to push data (e.g. via shared memory) to the other threads.
Probably a better design is to use unique processes (not threads) to do as much independent parallel work as possible until the processed data have to be joined, at which point you could use shared mem or another IPC to get them together.
While it's annoying this doesn't work, and I would welcome this being fixed, I have to admit that fixing pylsl in threads is fairly low priority. Due to the GIL there is barely any benefit to using threads in Python. If I have something I want to do in parallel without locking up the main function then I use multiprocessing.

Can you share a little bit more of your code, e.g. the read_stream module?

@cboulay the problem with using a single thread for multiple streams is that it introduces a lot of latency because it has to successfully pull in a sample from one stream before it can go onto the next, which completely locks up everything in case you have a marker stream you're pulling from. I guess I could try giving multiprocessing a shot instead.

@agricolab
def read_stream(stream_type):
    streams = resolve_stream('type', stream_type)
    inlet = StreamInlet(streams[0])

    ch_labels = []
    ch = inlet.info().desc().child("channels").child("channel")
    for k in range(inlet.info().channel_count()):
        ch_labels.append(ch.child_value("label"))
        ch = ch.next_sibling()

    csv_file = open(f'{stream_type}.csv', 'w', newline='')

    with csv_file:
        writer = csv.writer(csv_file)
        writer.writerow(["Timestamp"] + ch_labels)

        while True:
            sample, timestamp = inlet.pull_sample()
            csv_sample = [timestamp] + sample

            writer.writerow(csv_sample)

            sample_dict = OrderedDict(zip(ch_labels, sample))
            print(stream_type, timestamp, sample_dict)

Hello,
If it helps, I was also having issues with multiple threads on python, but only when I was trying to access inlet's info. I think the issues comes from how XML is handled internally by liblsl. I managed to find a workaround by parsing it manually, e.g.:
import xml.etree.ElementTree as ET
root = ET.fromstring(inlet.info.as_xml())
var_value = root.find("desc").find("var").text

(tested with python 2, run on python 3 but I did not try to reproduce the original problem)
Beside that I never had any troubles with multi-threading and LSL, that I have been using heavily :)

@cboulay the problem with using a single thread for multiple streams is that it introduces a lot of latency because it has to successfully pull in a sample from one stream before it can go onto the next

The preferred approach would be pull_chunk with a very small timeout so an irregular stream doesn't block anything.

If it helps, I was also having issues with multiple threads on python, but only when I was trying to access inlet's info. I think the issues comes from how XML is handled internally by liblsl. I managed to find a workaround by parsing it manually, e.g.:

import xml.etree.ElementTree as ET
root = ET.fromstring(inlet.info.as_xml())
var_value = root.find("desc").find("var").text

(tested with python 2, run on python 3 but I did not try to reproduce the original problem)

Interesting, is there any chance you could put together an MWE I could test against?

I never had any troubles with multi-threading and LSL, that I have been using heavily :)
Me neither. That's why i'm wondering.

I'l  assume you implemented the multithreading similar as follows?
t = threading.Thread(target=read_stream, args=(stream_type))
t.start()

The error probably stems, as @jfrey-xx  suggested from the info structure being accessed in a loop, considering it has a state due to ch = ch.next_sibling() and every thread might use the same dll. Maybe a lock around accessing the info until it is completely parsed might do the trick?

The preferred approach would be pull_chunk with a very small timeout so an irregular stream doesn't block anything.

@tstenner seems like a hacky approach which would still introduce unnecessary latency.

I'l assume you implemented the multithreading similar as follows?

@agricolab that is correct.

Something like
def read_stream(stream_type, lock):
    streams = resolve_stream('type', stream_type)
    inlet = StreamInlet(streams[0])

    ch_labels = []
    with lock:
        ch = inlet.info().desc().child("channels").child("channel"  )
        for k in range(inlet.info().channel_count()):
            ch_labels.append(ch.child_value("label"))
            ch = ch.next_sibling()

    csv_file = open(f'{stream_type}.csv', 'w', newline='')

    with csv_file:
        writer = csv.writer(csv_file)
        writer.writerow(["Timestamp"] + ch_labels)

        while True:
            sample, timestamp = inlet.pull_sample()
            csv_sample = [timestamp] + sample

            writer.writerow(csv_sample)

            sample_dict = OrderedDict(zip(ch_labels, sample))
            print(stream_type, timestamp, sample_dict)

https://www.bogotobogo.com/python/Multithread/python_multithreading_Using_Locks_with_statement_Context_Manager.php
On a side note: If you only pull single samples, but print and write to file every time, this might be slower than new samples coming in. You might want to switch to chunks or fill a buffer instead, unless your sampling rate is very slow.

@agricolab I guess that might do it.

On a side note: If you only pull single samples, but print and write to file every time, this might be slower than new samples coming in. You might want to switch to chunks or fill a buffer instead, unless your sampling rate is very slow.

I was aware of this and am going to rewrite it in the near future but thanks anyways.

Does inlet.pull_sample() in a thread definitely not block the GIL? Sorry I haven't tried myself.

Does inlet.pull_sample() in a thread definitely not block the GIL? Sorry I haven't tried myself.

Yes. Just tested it. Even if one of the outlets is killed, the others will not be blocked.

@jdevoldere , were you able to solve your issue, so it can be closed?

@agricolab , would you be willing to update https://github.com/labstreaminglayer/liblsl-Python/blob/master/README.md#known-issues to specify the problem with accessing info in a thread?
After that then I think we can close this issue.

Sure, i just found the time to test again pulling from different threads with Python 3.7.6. on Pylsl 1.14 with
from threading import Thread

def publish():
    outlet = pylsl.StreamOutlet(
        pylsl.StreamInfo("Mock", "Mock", 1, 100, "float32", "mocking")
    )
    count = 0.0
    t1 = pylsl.local_clock()
    print("Started publishing at {}".format(t1))
    while True:
        count += 1
        outlet.push_sample([count])
        while (pylsl.local_clock() - t1) < 0.1:
            pass
        t1 = pylsl.local_clock()

def subscribe(name: str, thread_name: str):
    inlet = pylsl.StreamInlet(pylsl.resolve_byprop("name", name)[0])
    print("Started subscription at {}".format(pylsl.local_clock()))
    while True:
        sample, timestamp = inlet.pull_sample()
        print(thread_name, sample[0])

if __name__ == "__main__":
    print(pylsl.library_version())
    Thread(target=publish).start()
    Thread(target=subscribe, args=("Mock", "Thread_1")).start()
    Thread(target=subscribe, args=("Mock", "Thread_2")).start()

and it worked smoothly.
@cboulay I can test whether the error with accessing the info stems from unlocked calls from different threads, that'll take longer to evaluate.

import pylsl
from threading import Thread, Lock
from time import sleep

def publish():
    channel_count = 8
    info = pylsl.StreamInfo(
        "Mock", "Mock", channel_count, 100, "float32", "mocking"
    )
    channels = info.desc().append_child("channels")
    types = (f"MockEEG" for x in range(1, channel_count + 1, 1))
    units = ("au" for x in range(1, channel_count + 1, 1))
    names = (f"C{x:03d}" for x in range(1, channel_count + 1, 1))
    for c, u, t in zip(names, units, types):
        print(c, u, t)
        channels.append_child("channel").append_child_value(
            "label", c
        ).append_child_value("unit", u).append_child_value("type", t)
    outlet = pylsl.StreamOutlet(info)

    count = 0.0
    t1 = pylsl.local_clock()
    print("Started publishing at {}".format(t1))
    while True:
        count += 1
        outlet.push_sample([count] * channel_count)
        while (pylsl.local_clock() - t1) < 0.1:
            pass
        t1 = pylsl.local_clock()

def subscribe(name: str, thread_name: str, lock: Lock):
    inlet = pylsl.StreamInlet(pylsl.resolve_byprop("name", name)[0])
    if lock is not None:
        lock.acquire()
    ch_labels = []
    ch = inlet.info().desc().child("channels").child("channel")
    for k in range(inlet.info().channel_count()):
        ch_labels.append(ch.child_value("label"))
        ch = ch.next_sibling()
    if lock is not None:
        lock.release()

    print("Started subscription at {}".format(pylsl.local_clock()))
    print("My channel has ", ch_labels, "channels")
    while True:
        sample, timestamp = inlet.pull_sample()
        print(thread_name, sample[0])

if __name__ == "__main__":
    print(pylsl.library_version())
    Thread(target=publish).start()
    sleep(1)
    lock = Lock()
    Thread(target=subscribe, args=("Mock", "Thread_1", lock)).start()
    Thread(target=subscribe, args=("Mock", "Thread_2", lock)).start()

accesses the XML in a locked fashion, and runs smoothly (on my machine)


Issue with azcopy to upload entire directory to storage drive

I keep getting the same error
"failed to parse user input due to error: the inferred source/destination combination is currently not supported. Please post an issue on Github if support for this scenario is desired"

My goal is to create a python script that will automatically transfer data when I run it. The SAS that I am using is not expired and I am unsure of the issue. Everything I have commented out is something I have tried to fix it.
My Code:
import subprocess
import os
import sys
#Works oncommand line by manually writing command, but not in program.
try:
    #key = input("Please insert SAS: ")
    key = 'SASKEY'
    #file_path = input("Please insert the full file path where the data is located: ")
    container = "container_name"
    #exepath = "azcopy"
    local_dir = "/dir/subdir"
    endpoint = "https://myblob.blob.core.windows.net/"+container

#Subprocess that performs data transfer to storage from the desired path
    #myscript="azcopy" + " cp " + "\""+ local_dir + "\" " + "\""+endpoint + key + "\"" + " '\"'"
    #print(myscript)
    #subprocess.call(myscript)
    #command = 'azcopy'+' cp '+'\"'+local_dir+'\" '+'\"'+endpoint+key+'\" '+'--recursive=true'
    #print(command)
    #print(['azcopy', 'cp','\"'+local_dir+'\"', '\"'+endpoint+key+'\"', '--recursive=true'])
    #exit(0)
    #subprocess.call(command)
    subprocess.call(['azcopy', 'cp', '"/home/dir/subdir/"', '"https://myblob.blob.core.windows.net/container?[SASKEY]"'])#,'\"'+local_dir+'\"', '\"'+endpoint+key+'\"', '--recursive=true'], stderr=subprocess.STDOUT)
    print('Transfer Complete.')

#Exception for exit error
except subprocess.CalledProcessError as e:
     raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
#Exception for EOF error, would be caused by missing token
except EOFError as error:
    print('Error with token')
#When an unexpected error has occured.
# except Exception as e:
#     print(str(e) + 'Unknown error has occured')
exit(0)

Also included a screen capture of code

Hi @buzzkillionair - Thank you for your feedback. Since this channel is more for driving improvements towards MS Docs, could you provide us with a URL of the documentation that your feedback is most applicable to? That way, we're able to locate the right team that's better suited to address your feedback.

@buzzkillionair Just following up on my previous comment above, please let us know if you were following a specific MS Doc so we can correlate your feedback to the right team.

@buzzkillionair Since we haven't heard back, we will now proceed to close this thread for now. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


[SPARK-10351][SQL] Fixes UnsafeRow.getUTF8String to handle off-heap backed UnsafeRows

CC @rxin @marmbrus

Merged build triggered.

Merged build started.

Test build #41791 has started for   PR 8523 at commit a967eb8.

Actually then it looks like the proper fix should make fromAddress handle base = null. Are there existing code that would break from that?

Test build #41791 has finished for   PR 8523 at commit a967eb8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41791/
Test PASSed.

Merged build finished. Test PASSed.

AFAIK only callers of fromAddress are UnsafeRow and UnsafeArrayData, both of which do null checking prior to calling.

Merged build triggered.

Merged build started.

Test build #41814 has started for   PR 8523 at commit 198daa7.

Test build #41814 has finished for   PR 8523 at commit 198daa7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41814/
Test PASSed.

Thanks - I've merged this.


Add OpenJDK 12/13/14 to Travis

Thanks for the contribution. 2 things:

The ASM library should be upgraded to support Java 13  & 14.
Gradle 5.2.1 doesn't seem to support Java 14: https://github.com/gradle/gradle/issues/10248

@chanseokoh thanks for the pointers. I will update PR

Oh, and seems like Error Prone has issues with Java 12+. Unclear if they are all fixed.

https://github.com/google/error-prone/pull/1107

Yeah, unfortunately, looks like we cannot add 12+ builds until they support them.

Oh, and seems like Error Prone has issues with Java 12+. Unclear if they are all fixed.

Nothing is fixed. I am struggling for 2 years now to submit pull requests that are ignored by error_prone developers, see https://github.com/google/error-prone/issues/1106
For JOSM we apply around 5/6 patches to make it work with recent versions of Java.

@chanseokoh gradle 5.x only works for 12-
this morphed into something else. Travis build is green
@don-vip maybe time to look for another tool ...

@gliptak for now we manage to update the tool with the patches. Maybe one day we'll drop it if Google continues to ignore recent Java. Are you working at Google, or just using their tools?

@don-vip just run into this while trying to bump Java version for jib

@don-vip maybe try the mailing list? https://groups.google.com/g/error-prone-discuss
@loosebazooka are we okay to upgrade Gradle, or it's safer to stick to the current 5.2.1 to prevent accidental API usage?

I mean I would like to stick to 5.2.1 if we could. Which part of this PR requires gradle 5.6.4? Is it the asm upgrade or the errorprone upgrade?

@chanseokoh I'm not following the "accidental API usage" concern
@loosebazooka the Gradle upgrade was in preparation for adding current JDKs

We try to stay on as low a gradle version as possible so we don't use API from newer versions of gradle -- If XYZ is only in gradle 5.6, we break all users who are on a version lower than 5.6. We have some testing for this, but it's not super comprehensive.
So staying at 5.2.1 keeps us compatible with more users.  We generally try to give users a year for moving up gradle version.  However there is nothing keeping the user from using 5.6.X in their build -- that should be fully compatible with anything we use from 5.2.1

Isn't the point of https://docs.gradle.org/current/userguide/gradle_wrapper.html that the pinned Gradle version is used (and automagically downloaded if needed) regardless of what Gradle (if any) is already installed by the user?

Ah so I think you're misunderstanding here. When building a gradle plugin you build with some version of the gradle api (in jib's case 5.2.1), however the gradle API is a provided dependency as is not packaged with the project, it's whatever is available at runtime -- so as long as user who has jib applied on their project is using gradle 5.2.1 or higher, it will work.
Pinning the version on the jib build creates no inherent restrictions on the version of gradle the user uses to build their project except for API compatibility and the code we have here: https://github.com/GoogleContainerTools/jib/blob/master/jib-gradle-plugin/src/main/java/com/google/cloud/tools/jib/gradle/JibPlugin.java#L85
So if the user needs 5.6.2 to build java 12 or 13, that's fine, we may not need to make this build use 5.6.2 unless jib introduces some explicit support for java 12 or 13.

@loosebazooka thank you for the explanation. Remove Gradle bump from PR

@gliptak thanks for check these! Merging.


Final cleanups from Readme

The following comments were added by @mattmiller899 in the readme. Let me know if this is still to be sorted. I'm removing these lines from the main readme file.

TODO: files that need paths fixed: bash/edit_interproscan.sh bash/run_start_lookup_server.sh

TODO: finish cleaning up pipeline/pipeline.py and pipeline/qc_pipeline.py TODO: finish clearing out old configs and other old files TODO: export the conda envs to ymls when finished

Thanks for your last commit @mattmiller899 I think we're now safe to close this issue.


Improve handling of temporary files

Codecov Report
Attention: Patch coverage is 85.18519% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 84.83%. Comparing base (540a3e7) to head (6d73e26).

Files
Patch %
Lines

src/hictk/balance/balance.cpp
77.77%
4 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #152      +/-   ##
==========================================
+ Coverage   84.79%   84.83%   +0.03%
==========================================
  Files         117      117
  Lines       10052    10047       -5
==========================================
- Hits         8524     8523       -1
+ Misses       1528     1524       -4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Auto-Completion from Language Schemas

Vega Editor has auto-completion for both languages. This is because Monaco editor (the editor used internally to Vega Editor and VS Code) has this baked in. Unfortunately, it is a technical marathon to get Monaco working inside a custom visual due to sandboxing, and to mitigate this will require the packaged visual to massively increase in size.
By replacing the editor, doing the JSONC changes (in #333) and adding the VS Code language APIs, we can write a custom auto-completion hook for Ace Editor that will mimic the same functionality for little additional cost.

This has been delivered to the alpha channel in its initial form. Given the size of the PR, we will create and track anything to do as new issues (or reopen this if absolutely necessary).


concise view doesn't continue to correctly report status after a retry event

Which version of kubefirst are you using?
v2.3.7
Which cloud provider?
DigitalOcean
Which DNS?
Cloud ones (default)
Which installation type?
UI (Console app)
Which distributed Git provider?
GitHub
Did you use a fork of gitops-template?
No
Which Operating System?
macOS
What is the issue?
i ran a mgmt provision that failed because of cloud quota, couldn’t create bucket in digitalocean because we ran out of space. the ui nicely offered me a retry, and the retry worked and provisioned the cluster. the problem is that the concise view never started back up. the detailed view continued, all the way to cluster complete, but the user was never offered the information that we were done in the concise view, and never was given the kbot password as a result.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

based off discussion with team we have rectified the issue as of now. Will revisit this issue if it pops up again


Configure automatic run of R based climate obs data prep

an R based shiny app has been created to review / fix / the climate observations data.  Need to:

identify the code used to prep the Xl Spreadsheet
Modify the paths
Containerize
Integrate the running of this code as a GHA.

Significant amount of work to accomplish this task.  Another task will also be defined that will port the existing R based Shiny app to run on openshift.

Had a thought that it may be possible to run the original XL code more easily.
Idea:

open the spreadsheet using python and the win32com module on windows vm
identify and modify the cells that describe the source paths for the incomming data
modify those paths to where the data is actually located.
Attempt to run the macro from python that performs the injestion.

Created the following branch with the xl code in it.  Its looking entirely possible.  Potentially can re-use this pattern for all the input spreadsheets that go into the coffee model.
Branch with the code is: https://github.com/bcgov/nr-rfc-climate-obs/tree/feat/15-xl-ingestion

Latest commit now updates the paths to the source data, however it does not update the hyperlink for the cell... Need to figure that out.
Also now updated the export models now get trimmed to only coffee and ensemble V

file path hyperlinks now getting updated.
Tmrw work on surrounding data imports:

create / pull weekend data, and re-run the import to see if it works.
also run todays data and import it.
start with fresh ss.

Next look into running as gha

working on getting the preparation of the wildfire weather stations (F_WX) data working and into directories / formats / file names that work with the xl injestion process.  Currently running into issues with getting the pulled data and the R script to work together

Shift is stragegy:

will pull the climate obs data down from their remote servers and do any transformations required.
climate obs data prep spreadsheet will be pulled from object storage.
update process will be run on the SS, which changes the default paths for where the input data is expected to come from
update process will run the macro
Run the coffee model (NOT PART OF THIS TICKET)

In order to enable this all to take place a sync process needs to be setup that watches the on prem versions of the spreadsheet and pulls that version of the SS.  Currently setting up a sync process that will copy updates of the SS from on prem to object storage so that that data is available for this process.  The process will be deployed using the GIS jenkins server

On prem -> Object store sync

Defined a jenkins job to perform the sync
Discovered the service account doesn't have access to the model path where the climate_obs excel ss is located.
Tried running the jenkins job under my idir (domain controller doesn't allow this)
Removed credentials from the job and reverted back to original
Created the following issue to track the service account creation: https://github.com/bcgov/nr-rfc-climate-obs/issues/42

the work associated with this ticket has been merged with main (https://github.com/bcgov/nr-rfc-climate-obs/pull/59), however the github action that was intended to run the climateobs xl macro is not enabled.  the GHA is https://github.com/bcgov/nr-rfc-climate-obs/blob/main/.github/workflows/run_climate_obs.yaml however none of the triggers will fire.
Moving this ticket to cancelled as we have pivoted our approach to try to re-create logic in the xl macro as a python script


3.0.0-1.2.0

Added

Added setup.sh script for bootstrapping the project.
Added insertTarget and insertLocation options to cf_notifier plugins
Added box-sizing-polyfill to exportsOverride as needed for
grunt-bower-task to work correctly. box-sizing-polyfill
is installed by cf-grid.
Added grunt watch:js task for completeness.
Added vendor directory variable to main.less.
Added warning for concat:cf-less Grunt task when sourcefiles are missing.
Added form for Submit a request FOIA page
Added styles, JavaScript for hiding and showing additional fields in forms
Added toplevel navigation items config file for removing hardcoded
navigation menu items.
Added external url redirect page, styles, and JavaScript.
Added .nav-secondary_link__disabled style.
Added .nav-secondary_item__child class to visually distinguish sub-pages
from sibling pages in the sidenav.
Added .nav-secondary_item__parent class to visually distinguish browse
pages from the subpages below them in the sidenav.
Added JavaScript utilities for checking types and primitives.
Added primary_nav jinja block to base.html template.
Added FAQ processor and mapping
Added use_form field to sub_pages
Added related_faq field to sub_pages and offices
Added inset-divider class for providing an inset horizontal rule
independent of the list or list item widths within the side navigation bar.
Added preview_text and short_title fields to sub_pages.
Added templates/activities-feed.html HTML template for the activity feed
area on the offices and sub_pages.
Added Plain Writing Feedback form.
Added cfpb_report activity type to activities feed macro.
Added breadcrumbs macro and temporarily set breadcrumbs for all office sub-pages.
Added download icons to privacy-impact-assessments-pias

Changed

Relaxed ESLint key-spacing rule to warning.
Refactored breakpoint-handler module into separate class modules
and utility function.
PascalCase ContentSlider module to properly designate class status.
Reduced complexity of validation and notification plugins
Changed vendor directory to src/vendor and updated paths.
Changed to using jit-grunt in place of load-grunt-tasks.
Updated contact us filter to use new notifications
(replacing type-and-filter messaging with cf_notifier)
Replaced placeholder Activity Feed on FOIA faq page with actual Activity Feed
Sped up notification animations
Added custom template for FOIA records page.
Refactored code for Wordpress updates
Initiatives renamed to Sub-pages
Relaxed ESLint cyclomatic complexity rule to max 4 complexity.
Updates megamenu bureau title to "The Bureau" to fit with sitemap direction.
Moved Less files to /src/static/css/ directory.
Updated cf-icons to 0.6.0.
Update processors.py for FAQ
Moved HTML templates to /templates/ subdirectory.
Breaks header template apart into header.html
and primary-nav.html templates.
Moved external site page header to its own template
header-without-nav.html.
Minor codefixes on show-hide-fields.js along with changing a class name for hiding fields
Updated side navigation bar to keep page order at mobile sizes and adds
"IN THIS SECTION" header text to the navigation bar dropdown menu.
Updated processors to use Elasticsearch bulk indexing
Office and sub-pages activity feed title to "Latest Activities"
and contacts to "Contact Information."
Moved activity_snippets macro from post-macros.html to macros/activity-snippet.html
and adds render method.
Made activity_snippet macro private.
Moved category_icon macro from post-macros.html to macros/category-icon.html
and adds render method.
Moved string_length macro from macros.html to macros/util/text.html.

Fixed

Fixed an issue where scripts were being initialized out of order
Fixed most of the warnings in validation and notification plugins
Fixed processor name bug
Fixed template/processor bugs while indexing and rendering
Fixed FOIA pages from the template/processor changes
Fixed missing states from .nav-secondary_link__disabled class for
visited and active links.

Removed

Removed copy:static-legacy and grunt-contrib-copy package.
Removed unneeded entries from exportsOverride in bower.json.
Gitignored CF fonts, "chosen" images, and other vendor files from repo,
which are slated for eventual removal.
Removed unused nav-secondary.html template.
Removed unused cf_inputSplit.js js module.

:+1:

:arrow_up:


[svclb-traefik] can't create /proc/sys/net/ipv4/ip_forward: Read-only file system

Environmental Info:
K3s Version:

k3s version v1.20.4+k3s1 (838a906a)
go version go1.15.8

Node(s) CPU architecture, OS, and Version:

Linux k3s 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Linux rk322x-box 5.9.14-rk322x #20.11.3 SMP PREEMPT Fri Dec 11 21:46:56 CET 2020 armv7l GNU/Linux

Cluster Configuration:

1 master = DigitalOcean droplet (2 GB Memory / 25 GB Disk / FRA1 - Ubuntu 20.04 (LTS) x64)
1 worker = Bare metal ARM machine, (2 GB RAM, Armbian 20.11.6 Buster)

Describe the bug:
Cluster works but the load balancer does not. It responds with 404 or 503 to external requests if ingress or service resources do not exist (this is normal), but it responds with 500 if they exist, and the request never reaches the service pods.
This is the logs from every load balancer pod / container:
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s <IP_ADDRESS>/32 -p TCP --dport 80 -j DNAT --to <IP_ADDRESS>:80
+ iptables -t nat -I POSTROUTING -d <IP_ADDRESS>/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

Running sysctl net.ipv4.ip_forward on all nodes shows net.ipv4.ip_forward = 1 so it's not clear what the problem is.
Steps To Reproduce:

Installed K3s as root using the quick start guide, everything on default
Worker machine is not necessary to reproduce, the load balancer gets stuck even on 1 node cluster, but I added it to rule out possible restrictions related to VPS on DigitalOcean.

I think the ip_forward error is not the actual problem. As you noted, ip_forward is already enabled, and the iptables rules are being created successfully.
Can you show the output of kubectl get service -A -o wide? Are you able to curl any of your target services from the nodes themselves?
Note that the default ServiceLB implementation isn't really suitable for distributed installations - it will assign the nodes IP addresses as the service's external IPs, and in this case the two nodes probably can't actually reach each other at their private IPs?

It's true that the worker node does not have the Flannel VXLAN port open, but I'm not planning to use the loadbalancer on all nodes and in any case, the problem is the same with a single node cluster.
I'm not sure what you mean by curling external IP. If you mean ClusterIP, yes, I can curl from the host the services on their clusterIP. Curling on the external IP, in my understanding, goes through the loadbalancer, and this is what I described it's not working. Here's the output you're asking:
NAMESPACE              NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE    SELECTOR
default                kubernetes                         ClusterIP      <IP_ADDRESS>       <none>           443/TCP                      4d4h   <none>
kube-system            kube-dns                           ClusterIP      <IP_ADDRESS>      <none>           53/UDP,53/TCP,9153/TCP       4d4h   k8s-app=kube-dns
kube-system            metrics-server                     ClusterIP      <IP_ADDRESS>   <none>           443/TCP                      4d4h   k8s-app=metrics-server
kube-system            traefik-prometheus                 ClusterIP      <IP_ADDRESS>   <none>           9100/TCP                     4d4h   app=traefik,release=traefik
kubernetes-dashboard   kubernetes-dashboard               ClusterIP      <IP_ADDRESS>   <none>           443/TCP                      4d3h   k8s-app=kubernetes-dashboard
kubernetes-dashboard   dashboard-metrics-scraper          ClusterIP      <IP_ADDRESS>   <none>           8000/TCP                     4d3h   k8s-app=dashboard-metrics-scraper
gitlab-managed-apps    certmanager-cert-manager-webhook   ClusterIP      <IP_ADDRESS>   <none>           443/TCP                      35h    app.kubernetes.io/instance=certmanager,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=webhook,app=webhook
gitlab-managed-apps    certmanager-cert-manager           ClusterIP      <IP_ADDRESS>    <none>           9402/TCP                     35h    app.kubernetes.io/instance=certmanager,app.kubernetes.io/name=cert-manager
kube-system            traefik                            LoadBalancer   <IP_ADDRESS>   <IP_ADDRESS>   80:31618/TCP,443:31275/TCP   4d4h   app=traefik,release=traefik

Meanwhile I created a sample nginx deployment/service and it works correctly with ingress and LB.
I think the problem is not k3s or the loadbalancer, because my problem only occurs when trying to access kubernetes-dashboard. It seems that the dashboard container only listens on httpS, and somehow this clashes with traefik expecting http.

I also encounter this problem. take the taefik service for instance.
# kubectl get service -A -o wide NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                                     PORT(S)                      AGE     SELECTOR default       kubernetes           ClusterIP      <IP_ADDRESS>       <none>                                          443/TCP                      2d23h   <none> kube-system   kube-dns             ClusterIP      <IP_ADDRESS>      <none>                                          53/UDP,53/TCP,9153/TCP       2d23h   k8s-app=kube-dns kube-system   metrics-server       ClusterIP      <IP_ADDRESS>   <none>                                          443/TCP                      2d23h   k8s-app=metrics-server kube-system   traefik-prometheus   ClusterIP      <IP_ADDRESS>    <none>                                          9100/TCP                     2d23h   app=traefik,release=traefik kube-system   traefik              LoadBalancer   <IP_ADDRESS>     <IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>   80:30373/TCP,443:32464/TCP   2d23h    #app=traefik,release=traefik

I also encounter this problem. Taking the traefik service for an instance
# kubectl get service -A -o wide
NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                                     PORT(S)                      AGE     SELECTOR
default       kubernetes           ClusterIP      <IP_ADDRESS>       <none>                                          443/TCP                      2d23h   <none>
kube-system   kube-dns             ClusterIP      <IP_ADDRESS>      <none>                                          53/UDP,53/TCP,9153/TCP       2d23h   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      <IP_ADDRESS>   <none>                                          443/TCP                      2d23h   k8s-app=metrics-server
kube-system   traefik-prometheus   ClusterIP      <IP_ADDRESS>    <none>                                          9100/TCP                     2d23h   app=traefik,release=traefik
kube-system   traefik              LoadBalancer   <IP_ADDRESS>     <IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>   80:30373/TCP,443:32464/TCP   2d23h   app=traefik,release=traefik

it should be accessible from all three external IPs displayed, however, it is actually accessible only from the external IP of the node where the pod is running, and not the other two.
I have ran the kubectl -n kube-system logs svclb-traefik-pl28k lb-port-80 command on each of the three svclb-traefik pods, and the output returned is as follows:
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s <IP_ADDRESS>/32 -p TCP --dport 80 -j DNAT --to <IP_ADDRESS>:80
+ iptables -t nat -I POSTROUTING -d <IP_ADDRESS>/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

I would like to know, should the service be accessible from the nodes other than the one where is pod is running, or the reported  error related to iptables are negatively impacting the access the service?


Fix to recursive SVD

Fixes #160. The fix used here is to use positive QR orthogonalization of the matrix V*S to recover both V and the singular values S. QR, based on Householder, is a much more numerically stable algorithm than Gram-Schmidt and by making the diagonal of R positive, computing QR for V*S gives Q=V and R=S.
Created a regression test to catch this inaccuracy issue if it happens again.

Codecov Report

Merging #167 into master will decrease coverage by 0.05%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #167      +/-   ##
==========================================
- Coverage   86.08%   86.02%   -0.06%
==========================================
  Files          39       39
  Lines        3363     3349      -14
==========================================
- Hits         2895     2881      -14
  Misses        468      468

Impacted Files
Coverage Δ

src/Tensors/linearalgebra.jl
89.02% <100%> (+1.35%)
:arrow_up:

src/Tensors/dense.jl
81.2% <100%> (ø)
:arrow_up:

src/decomp.jl
82.4% <100%> (ø)
:arrow_up:

src/Tensors/svd.jl
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e32f10...5369827. Read the comment docs.


submit the result of join-contracts.py to Etherscan by hand

To see if outputs from join-contracts.py are useful for Etherscan verification.

When I print the solc command line arguments from this line
I see
['solc', '--combined-json', 'abi,bin,bin-runtime,metadata,ast', 'raiden=contracts', 'test=contracts/test', 'services=contracts/services', 'contracts/EndpointRegistry.sol', 'contracts/SecretRegistry.sol', 'contracts/TokenNetworkRegistry.sol', 'contracts/Utils.sol', 'contracts/TokenNetwork.sol', 'contracts/Token.sol']

In raiden-contracts/raiden_contracts I can call join-contracts.py like
utils/join-contracts.py --import-map {} contracts/TokenNetwork.sol flat.sol

and the command finishes with status code zero, but the resulting flat.sol does not contain the definition of Utils.  I think this is a bug.   https://github.com/raiden-network/raiden-contracts/issues/522

When I call join-contracts.py with an imort map
utils/join-contracts.py --import-map '{"raiden": "contracts", "test": "contracts/test", "services": "contracts/services"}' contracts/TokenNetwork.sol flat.sol

the resulting flat.sol looks reasonable.

When I call etherscan_verify with TokenNetwork it succeeds but does nothing.  I should have tried TokenNetworkRegistry.  The scrips should show the list of valid inputs in this case. https://github.com/raiden-network/raiden-contracts/issues/528
python -m raiden_contracts.deploy.etherscan_verify --apikey <snip> --contract-name TokenNetwork --chain-id 3

join-contracts.py works on TokenNetworkRegistry.sol too, as expected.
utils/join-contracts.py --import-map '{"raiden": "contracts", "test": "contracts/test", "services": "contracts/services"}' contracts/TokenNetworkRegistry.sol flat.sol

I'm trying to compare the resulting flat.sol against the flattened.sol from the current implementation.

From raiden-contracts I called
python -m raiden_contracts.deploy.etherscan_verify --apikey <snip> --contract-name TokenNetworkRegistry --chain-id 3

and
diff raiden_contracts/deploy/flatten.sol raiden_contracts/flat.sol

I see differences around solc version pragmas.
1c1,2
< pragma solidity ^0.4.23;pragma solidity ^0.5.2;
---
> pragma solidity ^0.5.2;
>
23d23
< pragma solidity ^0.5.2;
64c64
< pragma solidity ^0.5.2;
---
>
105d104
< pragma solidity ^0.5.2;
155,156d153
< pragma solidity ^0.5.2;
<
1835,1836d1831
< pragma solidity ^0.5.2;
<
1919c1914
< }
---
> }

It seems like the current concatenation in etherscan_verify.py leaves many version pragmas in the result.  join-contracts.py leaves only one version pragma (correct).

Now I'm ready to try the result of join-contracts.py against Etherscan.
First I modify Utils.py a little bit (otherwise Etherscan already knows the Solidity source.  It will say: "Contract Source Code Verified (Similar Match)).
Then I compile the contracts by make compile_contracts.
I deploy the TokenNetworkRegistry (and others) by:
raiden-contracts$ python -m raiden_contracts.deploy raiden \
--rpc-provider <snip> \
--private-key <snip> \
--gas-price 30 --gas-limit 6000000

and I see an address on the console
    "TokenNetworkRegistry": "0xc48A9205745372A5bff6ebD5C6584920e784EC40"

I find it on Etherscan:
https://kovan.etherscan.io/address/0xc48A9205745372A5bff6ebD5C6584920e784EC40#code
and I don't see Solidity source (as expected).
I combine the Solidity sources locally
cd raiden_contracts
utils/join-contracts.py --import-map '{"raiden": "contracts", "test": "contracts/test", "services": "contracts/services"}' contracts/TokenNetworkRegistry.sol flat.sol

and submit it together with
ContractName: TokenNetworkRegistry
compiler: 0.5.2+commit.1df8f40c.Linux.g++
optimization: no

It seems that was successful.
I see some Solidity code on https://kovan.etherscan.io/address/0xc48A9205745372A5bff6ebD5C6584920e784EC40#code .


text/template: Better doc for ParseFiles and Execute behaviour.

Hello.
I recently spent some time trying to figure why a template with Funcmaps would work with Parse() but not with ParseFiles() when using Execute (instead of ExecuteTemplate with the name).
I found the answer in Stack Overflow http://stackoverflow.com/questions/10199219/go-template-function but the question also appeared in golang-nuts, for example here https://groups.google.com/forum/#!topic/golang-nuts/CI9WdC_sk00 and here https://groups.google.com/forum/#!topic/golang-nuts/vmZHHfLsH8M
Please consider if it should be better documented to make it less error-prone for inexperienced users.
Thank you.

I came here because I also wanted to open an issue about this. This API is not really developer friendly. Maybe one should change the semantics too and not only the docs. It feels wrong to name the most outer template again in template new just to add a funcMap via .Funcs.
// This is what currently works
tmpl := template.Must(template.New("layout.html.go").Funcs(funcMap).ParseFiles("tmpl/layout.html.go","tmpl/index.html.go"))

// This is what I would expect as a developer
tmpl := template.Must(new(template).Funcs(funcMap).ParseFiles("tmpl/layout.html.go", "tmpl/index.html.go"))

Maybe there are also nicer options but the current implementation doesn't really feel right and leads to confusion...


APM Agent Serialization Error

Describe the bug: APM Agent crashes when using Python AWS SDK's builder for DynamoDB Key condition expressions.
To Reproduce

Import from boto3.dynamodb.conditions import Key and make a DynamoDB query that includes KeyConditionExpression option with a value generated by Key class. Example Broken Code:

SESSION = boto3.Session(region_name="us-west-2",)
DYNAMODB_TABLE = SESSION.resource("dynamodb").Table("TableName")

options = {
    "KeyConditionExpression": Key("primary_key").eq(1)
}

result = DYNAMODB_TABLE.query(**options)

If APM Agent is running it will crash with the following exception when above code executes:

Exception in thread eapm event processor thread:
Traceback (most recent call last):
  File "/var/lang/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/var/lang/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/var/task/elasticapm/transport/base.py", line 145, in _process_queue
    buffer.write((self._json_serializer({event_type: data}) + "\n").encode("utf-8"))
  File "/var/task/elasticapm/utils/json_encoder.py", line 63, in dumps
    return json.dumps(value, cls=BetterJSONEncoder, **kwargs)
  File "/var/lang/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/var/lang/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/var/lang/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/var/task/elasticapm/utils/json_encoder.py", line 55, in default
    return super(BetterJSONEncoder, self).default(obj)
  File "/var/lang/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Equals is not JSON serializable

Excluding the usage of Key class for building Condition Expressions avoids the bug. Example Working Code:

SESSION = boto3.Session(region_name="us-west-2",)
DYNAMODB_TABLE = SESSION.resource("dynamodb").Table("TableName")

options = {
    "KeyConditionExpression": "#PrimaryKey = :value",
    "ExpressionAttributeNames": {"#PrimaryKey": "primary_key"},
    "ExpressionAttributeValues": {":value": 1}
}

result = DYNAMODB_TABLE.query(**options)

Environment (please complete the following information)

OS: Linux
Python version: 3.8
Framework and version [e.g. Django 2.1]: Not Applicable
APM Server version: 7.13.4
Agent version: 6.3.3

Thanks for the report. This should be fixed in #1203. May take us a bit to get it merged, I want to wait until @beniwohli is back from PTO to make sure there are no dragons lurking with a change like this.


Contract storage size panic on u32 overflow

Storage size is u32, and has value of all values size + storage_size_offset. If this exceed u32::max_value() then it currently panics.
Even if chain like Ethereum storage might never reach 4GB it might be the case for some peculiar chain using our framework. Thus I would go to make storage_size being u64.
This would imply some more complex conversion in rent computation but nothing that complicated.

storage size should remain u32; if anyone ever manages to store more than 4GB of data on-chain, and pays for it, then it's fine to just saturate the total at 2**32-1.

ok, implementation speaking it might not be easier because currently we only decrease/increase storage size on removed/inserted value. Thus we need to keep track of the real storage size otherwise a contract could go to a storage size of u32::max*2 and then remove u32::max size and he would have storage_size=0 but real_storage_size=u32::max.

ok, implementation speaking it might not be easier because currently we only decrease/increase storage size on removed/inserted value. Thus we need to keep track of the real storage size otherwise a contract could go to a storage size of u32::max*2 and then remove u32::max size and he would have storage_size=0 but real_storage_size=u32::max. or find another implementation

So switching from checked_add to saturating_add would make things worse? With the current code that panics on overflow we are in a situation where any extrinsic that increases the storage size above 4GB is simply invalid. This is usually bad because then no one pays for that invalid transaction. However, in order to make that happen the contract needs to pay a lot of rent.
That said, it might be worthwhile to switch from panic to a regular DispatchError.


Make modal action bar always shown in modal

http://stackoverflow.com/questions/28107637/div-with-max-height-containing-another-div-with-max-height-overflow-not-working

Duplicate of #536


Add ISO_3166-1_alpha-3 validator

According to wikipedia : ISO_3166-1_alpha-3 allows a better visual association between the code and the country names than the two-letter alpha-2 codes
(https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3)

Thanks. You're missing a lot of code here though. Could you check-in your src/lib/isISO31661Alpha3.js file? the file you've included is generated. Some tests and an update to the README would be appreciated too.

Done, sorry for the first bad commits

Thanks for the PR :smile:


[dv] update ram_ret partial write policy

addresses #2775

Signed-off-by: Timothy Chen<EMAIL_ADDRESS>

Feels like in an ideal world maybe we can consider generating the memory settings by decoding the hjson.
Although we would need to figure out how to do that for memories embedded inside peripherals...which is not something that is so clearly exposed at the moment...


Two together?

Acrolinx Scorecards
A minimum Acrolinx score of 80 is required.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology:

Article
AcrolinxScore
LowestCategoryScore
Issues
Scorecard
Processed

enterprise-graph/api-overview.md
78
61
10
link
:white_check_mark:

enterprise-graph/input-data-concepts.md
80
41
7
link
:white_check_mark:

enterprise-graph/ontology-overview.md
79
53
38
link
:white_check_mark:

enterprise-graph/ontology-tutorial.md
86
69
11
link
:white_check_mark:

enterprise-graph/ontology.md
87
39
3
link
:white_check_mark:

enterprise-graph/schema-map-tutorial.md
78
68
26
link
:white_check_mark:

More info about Acrolinx

Install Acrolinx locally for VSCode
False positives or issues

OPS Build status updates of commit ad53a0a:
:warning: Validation status: warnings

File
Status
Preview URL
Details

enterprise-graph/toc.yml
:warning:Warning

Details

breadcrumb/toc.json
:warning:Warning

Details

toc.json
:warning:Warning

Details

enterprise-graph/api-overview.md
:white_check_mark:Succeeded

enterprise-graph/input-data-concepts.md
:white_check_mark:Succeeded

enterprise-graph/ontology-overview.md
:white_check_mark:Succeeded

enterprise-graph/ontology-tutorial.md
:white_check_mark:Succeeded

enterprise-graph/ontology.md
:white_check_mark:Succeeded

enterprise-graph/schema-map-tutorial.md
:white_check_mark:Succeeded

enterprise-graph/toc.yml

Line 0: [Warning] Invalid file link:(~/ontology-concepts.md).

breadcrumb/toc.json

[Warning] Error happen when converting breadcrumb/toc.json to Pdf. Details: Could not find a part of the path 'T:\qw0x\breadcrumb\toc.json'.

toc.json

[Warning] Error happen when converting toc.json to Pdf. Details: Could not find file 'T:\qw0x\toc.json'.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


makeTranslationMatrix should be transposed

The Matrix is row-major so the translation part should be on ele[12], ele[13] ele[14]

I know that Matrix is column-major, but is there a reason why you think row-major?

I know that Matrix is column-major, but is there a reason why you think row-major?

Maybe I make a mistake, I found the matrix in fluid-engine-developmemt is row-major.

fluid-engine-dev is column-major, too.
template <typename T>
T& Matrix<T, 4, 4>::operator()(size_t i, size_t j) {
    return _elements[4 * i + j];
}

template <typename T>
const T& Matrix<T, 4, 4>::operator()(size_t i, size_t j) const {
    return _elements[4 * i + j];
}

Thx, I read the v1 code, it is different.
//!
//! \brief 2-D matrix class.
//!
//! This class is a row-major 2-D matrix class, which means each element of
//! the matrix is stored in order of (0, 0), (0, 1), (1, 0) and (1, 1).
//! \tparam T - Type of the element.
//!
template <typename T>
class Matrix<T, 2, 2> {
};

Oh, I confused row-major and column-major. fluid-engine-dev and CubbyFlow are row-major. 😭
Nice catch! I'll fix it ASAP. 😉
Reference: https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/geometry/row-major-vs-column-major-vector

I discussed this issue with Doyub Kim. (https://github.com/doyubkim/fluid-engine-dev/issues/323)

Please see my latest comment in the other issue in fluid-engine-dev. Hope that address the confusion.

@yangfengzzz I'll close this issue. Thanks. 🍡

Ok, I got it. With row major matrix and column vector, the code is correct. But I think it is better to use row major matrix with row vector or column matrix with column vector. Because when I use the matrix for Metal or OpenGL, the matrix have to be transposed then send to GPU.

@yangfengzzz Can you open another issue to refactor it?


Typescript support

Summary
Following the effort @falleco started on #71, this PR aims to bring updated types for v2.
🗺  Roadmap

[x] Write types for Server
[x] Write types for Server/Providers
[x] Write types for Server/Adapters
[x] Write types for Client
[ ] Try to compile types against existing project
[ ] Add TS example project
[ ] Add TS documentation

🚨   Disclaimer
I've been working with TS for a while, but this is my first time typing a library so feel free to comment extensively on what can be improved or might be incorrect. 🙏🏻
I'm also not familiar with typeorm or the inner workings of both adapters and provider so @iaincollins your guiding light 💡 here much appreciated 🤞🏻

Thanks @lluia, I really appreciated you picking up the work from @falleco for v1! I left the old PR both as reference for myself, and hoping someone might be able to help with it.
I just wanted to say it will probably be a few days before I can pick this up again, but it's really appreciated and it is something I would like to have.
One thing it would be really nice to have some developer tests or a script that we could run with an npm run command to help indicate what is missing types so we can improve that over time.
We already have linting with eslint, using StandardJS (npm run lint), if using a linter plugin is helpful there? Any recommendations on how we could do something like that would be really welcome. My reason for asking is that I would want to be sure we maintain them easily and they don't break over time when we change something and forget to update types.

Looking forward to this

@iaincollins after some investigation I decided to go with dtslint for linting the declaration files.
It seems is what most libraries are using 🧐 ( given is my first try doing this happy to move to other suggestions... )
Currently, it can't compile due to an error on next types ( which we depend on )
$ npx dtslint types
Error: At /Users/lluis.agusti/Code/next-auth/node_modules/next/types/index.d.ts:{"line":14,"character":6}: 'ts-ignore' is forbidden.
    at /Users/lluis.agusti/Code/next-auth/node_modules/dtslint/bin/index.js:198:19
    at Generator.next (<anonymous>)
    at fulfilled (/Users/lluis.agusti/Code/next-auth/node_modules/dtslint/bin/index.js:6:58)

I'm looking on how to bypass that 🤔 🤞🏻

@iaincollins after some investigation I decided to go with dtslint for linting the declaration files.
It seems is what most libraries are using 🧐 ( given is my first try doing this happy to move to other suggestions... )
Currently, it can't compile due to an error on next types ( which we depend on )
$ npx dtslint types
Error: At /Users/lluis.agusti/Code/next-auth/node_modules/next/types/index.d.ts:{"line":14,"character":6}: 'ts-ignore' is forbidden.
    at /Users/lluis.agusti/Code/next-auth/node_modules/dtslint/bin/index.js:198:19
    at Generator.next (<anonymous>)
    at fulfilled (/Users/lluis.agusti/Code/next-auth/node_modules/dtslint/bin/index.js:6:58)

I'm looking on how to bypass that 🤔 🤞🏻

Thanks! This seems like a good direction!
This is not an area I'm familiar with myself, but if we can get one or two examples working I'd be very happy to pitch in and try and improve (and maintain) coverage.

@iaincollins I opened an issue on dtslint to see if I can get help with the error above 💭
https://github.com/microsoft/dtslint/issues/297
I was thinking instead from writing the declaration files manually 🤔  , we could move the library to TS and let the compiler generate the types for us ( might be more scalable ). It'll involve though some work though 😆

I tried to start adding types to for the client/ directory but could only make it work by adding a client.d.ts file to the root of the repo (next to client.js).  How can I put these definitions within the types directory instead?

@arnodel I think the trick is to declare the module in this way:
declare module 'next-auth/client' {
  // type declarations here
}

in this way, it won't matter where declaration files are located ( as per TS docs ) and hence we can make it live inside of types/ ( keeping the project root cleaner in this way ) 💅🏻

^  Thank you to everyone working on this, I'm not commenting because I don't have anything to add to the convo but this is very much appreciated. 🙂

Not into TypeScript @iaincollins?

@LoriKarikari  I prefer strongly typed languages in general - C# is my all time favourite language - but I'm not personally a fan of TypeScript in practice, as I've run into too many issues with it (e.g. edge cases, compatibility issues, etc).
If pushed I would say I prefer Flow, however, I get that TypeScript is the most popular way of using types in JavaScript and a lot of folks get value out of it and I'm happy to support it.
I'm looking forward to seeing Deno evolve though! I think having better out of the box for types (fast and native) will make it a very different developer experience.

Thanks @lluia for the example.  However it doesn't appear to work for me with the new change - I cannot import from next-auth or next-auth/client.  I have attached screenshots from my IDE (vscode).  I must admit that although I know typescript well enough, the type declarations for javascript node modules are mysterious to me - so it is possible that I am under some basic misapprehension.

Previously those imports were resolved (with types/index.d.ts not containing a module declaration and client.d.ts at the module root).

@arnodel where are you trying to import those types? Currently they're not published hence not consumable on a project.
We'reusing TDSLint for now to help us write the declaration files which you can do through yarn lint:types. 🤔
Sent with GitHawk

@lluia I am importing from a project that uses next-auth :).   So let me check that I understand: currently the type declaration files are not exported, later they will be put in dist/, alongside the js files that they provide type declarations for?  I would find it useful to have that in place as I am currently using it.   Currently I am using the a workaround which consists of addings the following compilerOptions to the tsconfig.json of the project that uses next-auth:
   "baseUrl": ".",
    "paths": {
      "next-auth/*": ["node_modules/next-auth/types/*"]
    }

That doesn't seem a good long term solution to me however (no other package that I depend on requires that).
Also, when running yarn lint:types I got an error:
Error: /Users/adelobelle/personal/next-auth/types/client.d.ts:2:1
ERROR: 2:1  no-single-declare-module  File has only 1 module declaration — write it as an external module. See: https://github.com/Microsoft/dtslint/blob/master/docs/no-single-declare-module.md

The link says that declare module should "typically be avoided" - and to me it makes senses if the files are going to end up in dist/ as I describe above.
So perhaps it would be better to rewrite the contents of index.d.ts and client.d.ts without the declare module statements?  I have made it work by changing the compilerOptions/paths configutation of types/tsconfig.json slightly - see https://github.com/arnodel/next-auth/commit/3260060a2f68ea1fc1152cd4661205151041c624

@arnodel good catch, I think as you suggest using compilerOptions/paths for now seems a better approach than  declare module 💯  .
To clarify: for now, the types/* directory helps us keep the types in one place and lint them through dtslint. Later, once we're happy with the types, we will need to copy what's in types/*.d.ts to dist/ at build time, so we have them published on NPM and clients of this library can consume them.
If you need them straight away for a project you're working I'd suggest the best is to use npm link with this branch 🤔

@lluia personally I wouldn't mind having the type definition files in the src/ directory.  I feel like having them close to the js files makes the link more obvious, but I don't actually know what is generally done.  The dependencies I use seem to either be written in Typescript or have a separate @types package.
I also think it would be good to publish those types as soon as possible.  IMHO as this is a fork / PR, there is no need to worry about exposing a non-stable interface to unsuspecting users, but it allows people to use it easily (knowing the risks).
In my opinion lack of typescript definitions is likely to put off a fair number of potential users, so providing something usable early would be very valuable.

@arnodel the main purpose of types/ is for dtslint to work, however, we can always move it to @types 🤷🏻♂ , either way, the workflow is similar.
About publishing early versions, I don't think is a good idea to publish types half-way or not working. In case you need them ASAP, I suggest writing your  declaration file in your app or npm link meanwhile 🤔

@iaincollins @arnodel I guess moving them to @types feels more stable: if we submit a PR there with the types for this package then is gonna be reviewed by people with a lot of experience writing *.d.ts files and can advise us further.
I'm going to move what's being done here there for now, and then close then this PR ? 🤔

Update 🍉
@iaincollins @arnodel
I added types for what I think covers most of the library ( 👀 last commit ), the two APIs that are not fully typed are:

adapter ( typed with generic objects )

interface GenericObject {
  [key: string]: any;
}

type Adapter = (config: ConnectionOptions) => Promise<GenericObject>;

interface Adapters {
  Default: Adapter;
  TypeORM: {
    Adapter: Adapter;
    Models: GenericObject;
  };
}

callbacks ( typed with generic objects )

interface Callbacks {
  signin(profile: GenericObject, account: GenericObject, metadata: GenericObject): Promise<void>;
  redirect(url: string, baseUrl: string): Promise<string>;
  session(session: GenericObject, token: GenericObject): Promise<GenericObject>;
  jwt(token: GenericObject, oAuthProfile: GenericObject): Promise<GenericObject>;
}

Both these API accept tons of different configurations, I think is fair to leave them loosely typed for the time being and people interested in having more strongly typed can open news PRs.
What's Next 💭

[ ] Investigate whether to add these types as part of the build process or have them moved to DefinitivelyTyped.
[ ] Publish a beta version of the library with the types on it
[ ] Create a TS example repo consuming the beta and verifying all is fine

Nice work so far @lluia!

Hi @lluia is there anything I can do to help you with this?
Thanks for this PR!

@bamorim I think all types are ready ( some can be improved though, look for TODO in the changed files... ).
What's missing now is to write the tests and find the best way to publish them. I think ideally we would move this to DefinitivelyTyped.
Regarding publishing, we need to make sure we structure the files correctly given this library exposes four modules 📐
'next-auth' // server
'next-auth/client'
'next-auth/providers'
'next-auth/adapters'

As I understand the published types should then have this structure 💭
inde.d.ts // types for server
client.d.ts
providers.d.ts
adapters.d.ts

However, we need to play with the compiler to make sure it picks them up correctly. I'll try to give a push mid-week ( Wednesday ) but, any help is welcome especially if you have more experience typing libraries 👍🏻

What're your thoughts on migrating the codebase to TypeScript, instead of trying to mirror the types (and keep them in sync)?
I'm more than happy to PR, if you @iaincollins decide this is desirable.

@harrysolovay I suggested this here also, I think as an MVP we can publish the manual types on DefinitivelyTyped and in parallel open an issue and discuss how desirable ( and how much effort ) would be to move the source files to TS.
We need help testing the types written on this PR, so maybe it would a good place to start with your contributions 👍🏻

I agree with @lluia that the best steps are to see how far we get here, and to introduce tests to ensure we maintain coverage as we go. Getting to that point itself is not easy - though hopefully it is less of a moving target now 2.0 is out of beta - but it's premature to suggest we can handle migrating the entire codebase until we can at least get to that point.
Personally, I think that point has best trade off of effort vs benefit, and I am very happy to commit to support exporting types for API surfaces as I think it has clear benefits for users of the library and we can generally define well what request/responses we return (except in the case of raw objects from OAuth providers).

On moving the project to TypeScript
Migrating the actual project TypeScript should only happen after there are goals with defined benefits and understanding the tradeoffs - and crucially that means accepting that there are tradeoffs as well as benefits.
Certainly it's obvious what the benefits to users of the module are to have type definitions, but it would have been practically impossible to release this project if the whole thing were written in TypeScript (by any meaningful definition) because it would have multiplied the complexity and taken additional weeks or months to get to the same point, and I don't have time for that.
For sure using TS would almost certainly mean less bugs, and means spending less time on squashing them and the idea is that pays back dividends in future, but shipping with bugs you can fix later is better than never shipping.
If we had a larger / well funded / dedicated team that was up for supporting this in the long haul, I'd be open to taking a different view (as the cost/benefit changes); but the prospect of having someone start a re-write, lose interest and for us to reduce the number of people willing/able to collaborate on it while having increased LoC and overhead of maintenance is not appealing. Re-writes can really kill momentum on open source projects, because people have little incentive to push through the slog to get to working on the bits they feel deliver value.
Why would it be such a PITA on this project?
The main difficultly in this particular project that request and response objects from the services it integrates with are so varied and so conditionally and ambiguous that it's really hard to wrangle request/responses and you'd end up having to use any/object types all over the place or expect people to write a bunch of type definitions for each provider (ugh!) and provide a way for those type definitions to be conditional (as the same API endpoints return different responses under different situations).
How do we get to a better place?
I would suggest we get type definitions in (as @lluia suggests) and focus on supporting the tests that @JeffersonBledsoe has been working on, and getting to the point where can safely make more radical changes with confidence.
Once we have these two things - with a CD/CI pipeline that requires both pass before changes can be merged - we could revisit things then and decide on an order we might want to tackle them if we did want to start refactoring things in TS.
I would favour a gradual approach at that point rather than a big bang. I suspect it would take a long time to make the transition (maybe months, maybe a year or more) and that is something that would need to be factored in. Any time spent on this is time spent not adding new features or working on bugs, so the project may seem to stagnate, which is again another reason for caution.
I appreciate one dedicated person could do a rewrite in much shorter time, but it would still be several weeks worth of work and would involve a lot of tedious regression testing. It only took me ~6 weeks working in spare time (evenings, weekends and lunch breaks) to build v2  (with help from the other core team members), so it's possible, but not fun and I expect would take longer as you'd have to face up against challenges with database integration and provider definitions to implement types in any meaningful way (i.e. without falling back to any/object in key places and hoping for the best).
Personally, if someone asked me how long it would take me working on it full time to migrate the entire project to TS I'd estimate at least as long as it took to write in the first place and that it would probably take longer than that.

@iaincollins what about gradually transitioning to TS? All the JS keeps working anyway so it's not a complete rewrite from scratch. For me, it's fine if it just stays JS btw just giving my 2 cents.

@iaincollins I completely agree with everything you've said. I've created a draft nonetheless, so we can start hammering out specifics. I don't expect it to be merge-ready all too soon... but it's a starting point. More testing is gonna be critical to ensuring a smooth transition.

Update
@iaincollins @harrysolovay @LoriKarikari I opened a new PR on DefinitivelyTyped: https://github.com/DefinitelyTyped/DefinitelyTyped/pull/45775 with all the types updated.
Tests
This test file covers the types for the exported API of next-auth ( cc. @iaincollins ).
Hopefully, I didn't miss anything, but if so we can always publish and keep on iterating.
Owners
I marked myself and @iaincollins  as owners of the future @types/next-auth package, @harrysolovay let me know if you want to be added as well ( or anybody else too ). Once published, we need to sync on how to communicate best API changes on this package so types can be always up to date.
Follow up
I'll follow up there and leave this open for reference 👍🏻 ( we're almost there! 🎊 )

Great work @lluia! :100:  Sorry for not being that present. I'll be working on my Next.js project next weekend and then I can spare some time helping. (I'm on a really tight schedule between multiple side projects, so sorry about that).
@iaincollins I totally get your point. Better ship it and make it work, and then, once you have a stable enough library then migrating to TS is easier than prototyping in a lot of cases. It might not even be worth, as you said. But new versions of TS keep adding niceties to the type system like unknown added in 3.0 and the better recursive type alias added in 3.7. What I mean is that I feel like TS is trying to evolve to make those cases easier to represent. My point is that waiting may make this job actually easier just because TS will release new features.

@lluia what's happening with the typings now? I see that the CI is failing on DefinitelyTyped... is there anything I can do help out?

@samsamson33 I need to have a look, hopefully, this week  👀 🤞🏻 , I think the compiler is trying to parse the types from next and failing:
##[error]node_modules/next/dist/next-server/lib/router/router.d.ts(72,21): error TS2304: Cannot find name 'PopStateEvent'.

A workaround, for the time being, is to not relly on next types and type those manually. I'll give it a go 👨🏻💻 ( feel free to suggest any improvements or changes you see fit 💯  )

@lluia could not finding PopStateEvent possibly be because you don't have "dom" in your tsconfig's lib? (Sorry if I'm way off the mark here)

@samsamson33 ah good point 💥 , let me push it to the PR and see if it makes the error go away 🤞🏻

@iaincollins @lluia @samsamson33 @bamorim and others. awesome work on this! (I was disappointed that my original work on typings weren't updated for this release, so I came looking)
I spent an hour or two trying out converting next auth to typescript. I just have to finish typing the options and a few other small issues (and taking some types from this PR). @iaincollins would you be more open to accepting a PR like this? Most of it is utilizing type inference or any. but at the very least it's up to date and can be improved incrementally with (hopefully) no impact on existing users (js or TS), I've added minimal interfaces where it makes sense, and fixed a few type errors where necessary. I'd be curious to hear your objections and drawbacks of typescript (beyond initial development, because I think everyone agrees with that point)

@iaincollins @lluia @samsamson33 @bamorim See the PR I just submitted for a working migration to typescript. I'm sure there's more work to be done, so please check it out and provide feedback!
We might end up abandoning it, because of reasons explained earlier in this chain, but I thought I'd remove the "too much work to transition" hurdle. (because really its not hard... we could have stopped at just a rename and add a bunch of anys, but I wanted to do it right).

I favour @lluia approach in this PR, which is incremental and gives a chance to test things out as a team.
With regard to TypeScript, supporting pull request is my priority right now.
If anyone is looking for a way to help out to support a move to TypeScript (beyond helping out with this PR) the most useful thing right now would be to help out with the end to end tests, which have sadly been on hold for too long.

Update 🗓
So the types are finally ready on DefinitivelyTyped 💆🏻♂ 🎊 🤞🏻 💥
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/45775
I'll need now a couple of reviews (maybe @iaincollins and @NickBolles ?) to be able to  merge it 💚
I'm aware they're less than a perfect but I think is a good start and we can get them out and iterate later.
I think the key for them to grow strong is to sync well whenever changes happen in this library. As a first step making CHANGELOG.md would be good, but here's more 🥗 for thought.
Why it took so long ⏲
I was caught forever with a cryptic error related to 'source-map' that went away once I removed next types from the dependencies. This, however, means I'm typing NextApiRequest and NextApiResponse manually which seems a bit brittle and I'm eager to see how it plays once the types are released...
The fact also that I lost my job due to COVID and been actively looking for a new one since a few weeks made a bit challenging keeping up with open-source 😅

@m5r thanks for your comments 💯 , I'll try to get them addressed soon. Could you next time comment on the DefinitivelyTyped PR? The work is carried on there and this one is just open for reference.

Finally @types/next-auth is published, we can continue to raise PRs in DefinitivelyTyped to improve them 💚 💯

Great work @lluia!

@lluia Thank you and congratulations! <3
What are your thoughts on merging this PR into NextAuth.js also, so we can work towards better support for types natively?

are there any plans to add types for next-auth/jwt as well?

@iaincollins I think the types on this PR are a bit outdated, so in that case would be best to directly copy what we have on DefinitivelyTyped 🤔
I'd say for now lets see how @types/next-auth plays out, there's people making PRs on it  🎊 , too bad I'm on holidays this week 🏖 with limited access to internet, I wanted to find time to create the example TS repo 🤞🏽

I could do an exact copy of the example repo but in TypeScript. Just create an empty repo in the organization and I'll PR it

are there any plans to add types for next-auth/jwt as well?

@alxyuu, I did them. The issue is that I need some types from jose, and to do that, this PR has to be approved. Then, I will push the types for JWT!

@JuanM04 here you go if you have time 👍🏽 , would be cool to keep the file structure same as the default example repo.
In case you don't find time don't worry! I'll try to PR on it once I'm back from hols ⛱ next week ( @iaincollins )

@lluia Thanks for this contribution, and more importantly, enjoy your holidays! 😃

Thank you, @lluia! Enjoy your holidays!

I finished the repo, but some types are wrong. I opened a PR fixing them

@lluia Thanks for your feedback! Sounds good to me. Have a great holiday! :-)

great !! thx


Windows 8.1 issue installing mocha -g

After running npm install mocha -g it throws an error.
npm ERR! not a package C:\Users\homer\AppData\Roaming\npm\mocha
npm ERR! addLocal Could not install C:\Users\homer\AppData\Roaming\npm\mocha
npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "mocha" "-g"
npm ERR! node v4.4.2
npm ERR! npm  v2.15.0
npm ERR! path C:\Users\homer\AppData\Local\Temp\npm-25356-63c186ca\unpack-298d18e03c67\package.json
npm ERR! code ENOENT
npm ERR! errno -4058
npm ERR! syscall open

npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\homer\AppData\Local\Temp\npm-25356-63c186ca\unpack-298d18e03c67\package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! Please include the following file with any support request:
npm ERR!     D:\tmp\test-mocha\npm-debug.log

Looking inside:
C:\Users\homer\AppData\Local\Temp\npm-25356-63c186ca\unpack-298d18e03c67
I see only 1 file, index.js
I tried running from both power shell, and git-bash.
(I am able to install mocha to a local folder)

despite the error text claiming that it's not responsible, I'd report this to npm.

Updating to npm 2.15.9 fixed the issue.


[GLUTEN-2217][CH] Introduce Signal Handler to print stack info when get SIGSEGV

What changes were proposed in this pull request?
(Fixes: #2217)
How was this patch tested?
using exited UTs
Design
Experimental feature.

Codes are copied from BaseDameon, since I believe it's difficult to merge into the main branch.

The idea is very simple, register a single handler via Signal Chaining, when jvm failed to handle signal, it will call ours. We create a pipe during initialization, and in our Signal handler, we collect stack info, and then write those info into pipe. We also create a Signal Listener to read from that pipe, and dump the stack info.
In most cases, the method works as I tried out. However, if the original SIGSEGV issue results from a corrupted stack, such as a JNI stack overflow, our signal handler won't be able to work. I spent a lot of time figuring out why, but I made no progress.
For example, calling the following function in the JNI could easily reproduce the case.
void deepNative64K(Poco::Logger * log, int depth = 10000) {
    char BUF[256*1024];
    memset(BUF, 0x5D5D5D5DL, sizeof(BUF));
    LOG_WARNING(log, "thread({}), stack grows {} kbytes", getThreadId(), (sizeof(BUF) * (10001-depth))/1024);
    if (depth > 0) {
        deepNative64K(log, depth - 1);
    }
}

How to use
export LD_PRELOAD=libjvm.so-directory/libjsig.so:libch.so

LGTM


Pasted content is lost after char insertion, when using contentEditable on IE11

TO REPRODUCE:
Take any stock mode, and change the input type to contentEditable
In IE11 (tested on win 8.1), paste some content into an empty editor, then hit the space bar (or any key)
EXPECTED:
A space (or the character pressed) appears after the pasted content
ACTUAL:
The pasted content disappears! (the added char remains)

I know you've got a huge amount of work going on, so I hope this doesn't come across as pestering. I wonder, however, if there's a way I might up my monthly contribution as a way of upvoting this bug? As a consumer of this library, it's only reasonable that I pay for your time. smile

Could you see if attached patch helps?

It appears to fix things on my IE11 VM - let me do some more testing when I can get my hands on a real machine and post back in 24 hours. Thanks for looking into this!

No dice. :(
I've put up a test instance using the stock Scheme mode and a pull from https://github.com/codemirror/CodeMirror/commit/fafb7dc17d45d24ef5e84d3f61696ae9a4ea28fd. To reproduce (confirmed on IE11.0.9600.18376 and 11.494.10586.0):

Open the URL: http://www.bootstrapworld.org/preview/testing/codemirror2/mode/scheme/
Locate the line beginning with (define BACKGROUND (bitmap/url "..., and select the contents of the quoted string (the URL)
Open a new tab and go to any URL. Then copy the URL from the address bar
Return to the first tab, and paste the clipboard URL on top of the selected URL. Then hit the space bar.

Sometimes the pasted URL disappears, and sometimes it disappears the the old URL comes back.

I messed up my fix (and somehow, it looked like it worked when I tried it). Attached patch should make it actually do what it was supposed to do.

Confirmed fixed! Thanks so much. Not sure what the right etiquette is for closing, so I'll leave this open until you want to shut it down.

Great, I'll close.


[5.1] Fix URL generator port check

Fixes the port from being added to generated URLs when unnecessary. Symfony returns the port as a string and Laravel is doing a strict comparison with an integer.
This change caused :80 to appear in all of my generated URLs when the scheme was already http.
Edit: Digging more into the Symfony source code, it looks like getPort() can return an integer or a string.  The docblock specifies string, however.  I am using a proxy, which causes symfony to return $this->headers->get(self::$trustedHeaders[self::HEADER_CLIENT_PORT]) (a string).

This change is wrong. You need to keep the === and compare it to a string instead of an integer.

@GrahamCampbell No, because getPort() can return a string OR an integer. Take a look at the code here. It will not always be a string.


after update

[+] Authentication renewed!
[+] Authentication renewed!
{"error":"access unauthorized","code":6}

check main.py updated


Bump to build with go1.17

/kind cleanup
Build cert-manager with go 1.17

The controller shrinks a little:
# go 1.16
60M bazel-bin/cmd/controller/controller_/controller

# go 1.17
58M bazel-bin/cmd/controller/controller_/controller

At a guess this might need a change in the upstream go bazel rules first; I wondered if this was gonna be trivial and it's clearly not :grin:

closed in favour of #4381


cdk Table

I'd like to see a documentation of cdk table with pagination, sorting and etc without Material design. Can anyone guide me how to implement these functionalities without Material? Thank you.

These are the CDK table docs: https://material.angular.io/cdk/table/overview.

@crisbeto yes, I saw that, but it misses lotta stuff.  I couldn't find any docs about pagination or sorting on server side for example. Like the one an Angular Material table have. Can you help me to implement this functionalities ? I'm creating cdk table with bootstrap table styles.

You can look at mat-paginator and mat-sort-header as examples of how you could accomplish this. The CDK itself is agnostic to any particular design system; sort-headers and paginators are inherently tied into a certain look. However, the flow of data between these components and the data source is the same either way.


[#126965751] Add jq 1.5 to the cf-cli container

#126965751 Test users not removed from CC DB
What
We need to parse the output of cf curl /v2/users to list all the existing users created by the acceptance tests. For this jq is perfect.
We consider that this really useful and it will be used in other stories in the future, so we add it as a basic tool in the cf-cli container.
We want to use the regex based functions, so jq must be >= 1.5 and must be compiled with ONIGURUMA regex library. The Ubuntu package is 1.4, so we will pull the static binary released upstream.
Dependencies
Please merge #65 first and rebase.
This could be reviewed together with https://github.com/alphagov/paas-docker-cloudfoundry-tools/pull/62
How to test?
Travis shall pass.
You can test locally with rake build:cf-cli spec:cf-cli.
Then you can test: docker run -ti cf-cli jq --version
Who?
Anyone but @richardc or @keymon

Build failed to install json gem. I haven't investigated why.
I'm not totally against this change, but I'm unsure about the precedent it might set for shell scripts manipulating data that could benefit from being written in another language that supports data structures. Could you use Ruby (which is in the cf-cli container) to achieve what you're doing at the moment? Would it be much harder?

An equivalent ruby script is possible, but would be another small piece of software we need to write and maintain, something like:
document = JSON.load(STDIN.read)
document['resources'].each do |elem|
   username = elem["entity"]["username"]
   next unless username =~ /"^custom-acceptance-test-user/
   puts username
end

vs.
jq -r '.resources[] | .entity.username | select(. != null) | select(test("^custom-acceptance-test-user"))'

jq is approaching a the place of a standard 'awk for json' tool so I think it's safe to adopt.

Travis build is failing as the ruby in use has switched from 2.2.3p173 to 2.3.1p112.
2.3.1 is api incompatoible with the pinned version of the json gem.  Recommend pinning ruby version in the .travis.yml

Rebased after merging #65

@keymon I need to run bundle install to follow the manual testing instructions, and I can't use rbenv without a .ruby-version. I'm happy to add that as part of this PR, but what version should I use?

I guess your comment refers to https://github.com/alphagov/paas-cf/pull/430
I think we should discuss within the team if we should add .ruby-version in the repos. About version, I think we agreed on 2.2

@keymon my comment refers to the instruction You can test locally with rake build:cf-cli spec:cf-cli. in this PR.
Pinning the Ruby version is not really optional, but very happy to discuss tooling consistency within the team. I'll add a .ruby-version at 2.2 in the meantime. Thanks.

Alpine 3.3 uses ruby 2.3, instead of 2.2
I did not want to change such stuff.

You mean 3.4 uses 2.3? and 3.3 2.2?

builds and tests correctly with ruby 2.2.2. I have also pulled image built by docker hub and that one also looks OK, merging

the comments about versions belong to https://github.com/alphagov/paas-docker-cloudfoundry-tools/pull/62


FIX CSL 738

Hello!
We noticed a pink colour in the front of the B738 CSL of the newest VLC CSL.
Is that something we could fix?
The B738 needs winglets
Best regards and Happy new year!

Could you compare both?
The old one and the we think is working?
Thanks again!

Sure thing. It will have to be next week, very busy at the moment.

Done

Thanks!!


Release 1.9.0

coverage: 33.981%. first build when pulling 8df4b6b64b5237159a7705f0021c2439281d0011 on release_1.9.0 into d7c242859c1186484950f0f06692491c2b46e37b on master.


Return remappings inferred by the LocalNpmResolver in the CompileResult

The compileSol and compileSourceString() use the LocalNpmResolver class to infer any missing remappings to packages located under a node_modules directory in a parent directory of the current compiled artifact.
It would be useful to callers (especially scribble) if those entry points also reported in their CompileResult any additional rempapings that they inferred, on top of the remappings passed in by the user.

Codecov Report

Merging #109 (f0f3f56) into master (3ab8d4e) will decrease coverage by 0.02%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #109      +/-   ##
==========================================
- Coverage   91.95%   91.92%   -0.03%
==========================================
  Files         253      253
  Lines        5255     5266      +11
  Branches      834      836       +2
==========================================
+ Hits         4832     4841       +9
  Misses        262      262
- Partials      161      163       +2

Impacted Files
Coverage Δ

src/compile/import_resolver.ts
86.20% <75.00%> (-4.71%)
:arrow_down:

src/compile/utils.ts
93.02% <100.00%> (+0.22%)
:arrow_up:

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

I think you're right that there could be duplicates, so its a good decision to turn it into a map from files to their remappings. Its also strictly more information. Looks good to me as it is!


read_excel(sheet_name=None) not working

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.14.3
Modin installed from (source or binary): source
Modin version: 0.4.0
Python version: 3.6
Exact command to reproduce: df.read_excel('File', sheet_name=None)

Describe the problem
When loading an excel file using sheet_name=None, which loads in all sheets. The script crashes. The script works when using pandas directly, so that shouldn't be a problem...
Source code / logs
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
return handler(request, *args, **kwargs)
File "/Users/marcomilov/Thesis/reportAutomater/reportForm/views.py", line 41, in post
df_tup = self.fileService.extractRawSourceSheetFromExcel(file)
File "/Users/marcomilov/Thesis/reportAutomater/reportForm/file_service.py", line 41, in extractRawSourceSheetFromExcel
df = pd.read_excel(f, sheet_name=None)
File "/usr/local/lib/python3.6/site-packages/modin/pandas/io.py", line 222, in read_excel
return DataFrame(query_compiler=BaseFactory.read_excel(**kwargs))
File "/usr/local/lib/python3.6/site-packages/modin/data_management/factories.py", line 95, in read_excel
return cls._determine_engine()._read_excel(**kwargs)
File "/usr/local/lib/python3.6/site-packages/modin/data_management/factories.py", line 99, in _read_excel
return cls.io_cls.read_excel(**kwargs)
File "/usr/local/lib/python3.6/site-packages/modin/engines/base/io.py", line 340, in read_excel
**kwds
File "/usr/local/lib/python3.6/site-packages/modin/engines/base/io.py", line 13, in from_pandas
return cls.query_compiler_cls.from_pandas(df, cls.block_partitions_cls)
File "/usr/local/lib/python3.6/site-packages/modin/data_management/query_compiler/pandas_query_compiler.py", line 500, in from_pandas
new_index = df.index
AttributeError: 'collections.OrderedDict' object has no attribute 'index'

Hi @marcomilov, thanks for reporting this!
The change to fix this is relatively simple, we just need to add a conditional check based on the return type of the code that defaults to pandas, and return the proper OrderedDict instead. This line: https://github.com/modin-project/modin/blob/master/modin/engines/base/io.py#L313 is where that code should go.
I am labeling this as "Help Wanted" for the time. We have a backlog of other robustness changes and features we are working on, but I will make sure this will make it in the next release. Let me know if you are interested in contributing!

@devin-petersohn i try to fix this in https://github.com/modin-project/modin/pull/532 but i don't know if its correct. I couldn't use the modin_df_equals_pandas to check if the values in the returned dict of each implementation where equal (because the modin df in the value has no query_compiler.

Resolved by #532


SEO Issue: meta descriptions too short

Recently the docs team fixed an issue in order to populate every docs page with a unique meta description (thank you @jseldess). Which was really great.
But now we have an issue with some of those descriptions being too short. Here is a report: https://docs.google.com/spreadsheets/d/1r39xR3J35v9Vc01FuOLq2gLnFFaGa5PgnTsQyX8nMNc/edit?usp=sharing
Is it possible to require that each docs description be between 50–160 characters?
Context: this work is part of a larger SEO project to optimize the whole site. With the goal being to double our site traffic.

@dankellycockroach, just created an issue to check for the presence and length of meta descriptions: https://github.com/cockroachdb/docs/issues/10183. Can we close this in favor of that? Once that's in place, all the short descriptions should be caught and then fixed.


[D3D12] Add missing DLL architecture paths for the 32-bit builds.

32-bit paths were missing (both Agility SDK and DXIL have 32-bit builds, PIX do not).

Thanks!


fix dual-stack ipv6 listening on windows, add listen addr option

According to windows document, the sockets only operates over the IPv6 protocol by default, so we need to modify the socket before using it.
Also --listen-addr  option is added to be able to bind on a certain address, default is [::].
Tested on win10 and linux.
Refs:
https://learn.microsoft.com/en-us/windows/win32/winsock/dual-stack-sockets
https://docs.rs/tokio/1.29.1/tokio/net/struct.TcpSocket.html
https://docs.rs/socket2/latest/socket2/struct.Socket.html#method.set_only_v6

IPv6 has been an official internet standard since 2017, and thus IPv6 support should be added everywhere. I do not understand why your PR hasn't been merged yet...

IPv6 has been an official internet standard since 2017, and thus IPv6 support should be added everywhere. I do not understand why your PR hasn't been merged yet...

💩

Marking a helpful suggestion as spam 💩

I'm closing because additional options are not in scope for bore. It's a small program and meant to be read/forked if you have more particular use cases.
I don't have the resources to maintain a general-purpose tool, and I think something like this is better built for your specific use case regardless

why not supper v6


chore(deps): upgrade configuration

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-configuration-k8s-29-main" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 4418

Questions ?
Please refer to the Backport tool documentation


Wrong field name for custom id in create method

The body.key field is ignored, only id and uri are actually used:
let { id, uri } = body;

Thank you!


Break out cyhy-core install and retrieval of GitHub OAuth token into roles

Both these tasks are repeated in the Ansible code, so it makes sense to break them out into their own roles.

@dav3r, please take another look.  If you're still happy then I'll merge this pull request and start applying dependencies globally.


Benchmark should use a time source with sub-second resolution

The benchmark uses time() from time.h to measure elapsed time. It measures time with 1 second resolution. As certain runs complete in 6 seconds or less, the not so fine grained time source introduces significant inaccuracy.

I agree. I had chosen time() due to its portability, but it's not as good as it could be. Which timekeeping system would you recommend?


Notification Service Extension Setup (without Cocoapods) results in Undefined symbols for architecture

Description:
I've completed the React Native OneSignal setup guide, and getting an "Undefined Symbols for Architecture" error which looks like this:
Undefined symbols for architecture arm64:
  "_OBJC_CLASS_$_RCTConvert", referenced from:
      objc-class-ref in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
  "_OBJC_CLASS_$_RCTEventEmitter", referenced from:
      _OBJC_CLASS_$_RCTOneSignalEventEmitter in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
  "_OBJC_METACLASS_$_RCTEventEmitter", referenced from:
      _OBJC_METACLASS_$_RCTOneSignalEventEmitter in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
  "_RCTRegisterModule", referenced from:
      +[RCTOneSignalEventEmitter load] in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
  "_RCTRunningInAppExtension", referenced from:
      -[RCTOneSignalEventEmitter checkPermissions:] in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
      -[RCTOneSignalEventEmitter requestPermissions:] in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
      -[RCTOneSignalEventEmitter getPermissionSubscriptionState:] in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
  "_RCTSharedApplication", referenced from:
      -[RCTOneSignalEventEmitter checkPermissions:] in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
      -[RCTOneSignalEventEmitter requestPermissions:] in libRCTOneSignal.a(RCTOneSignalEventEmitter.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Notes about my setup:

Setup without Cocoapods using react-native link
Used yarn for installation
React Native Version: 0.57.7
Notification Service Extension written in Obj-C

The library works completely fine without the notification service extension, so it doesn't seem like it's a problem with the react-native link step. It also has no problems until the compilation stage, meaning the header search paths are working correctly. I've added the three libraries as instructed to the Notification Service Extension target. Would love some help here!
Environment

react-native: 0.57.7
react-native-onesignal: 3.2.12

Let me know if I can provide any more info!

It's funny how once you write something out you start to understand it... In trying to replicate the bug with a fresh react-native init MyAwesomeProject I noticed some differences between that project and my own...
In creating a new Target, many build settings from my main app's target were carried over unknowingly. Once of those being the -Objc flag, which after removing from my Notification Service Extension Target's build settings, fixed the issue. Hope someone learns something from this!

I was able to reproduce this issue, but the fix did not solve the problem in my test build. Others are running into this issue so while we investigate, I'm wondering if anyone knows of other ways to resolve / temporary workarounds. Thanks!


Honeypot other AP names?

Nice job you're doing here, I love the concept.
One feature suggestion that could be interesting:
Instead of manually entering a new AP name, would it also be an option to select an SSID name from a list of existing AP's and just have the device sitting there pretending to be this AP?

That is a really good idea, I like it a lot!
I'll update the list of to-do items today. Thank you for the feedback

I added this to the "Todo" section of the main readme. Thanks again!