v0.7.0 Pre-Release: More config options, LDAP/remote auth support, extractor updates, usability and performance improvements, bugfixes

DONE:

[x] pass --single-process --no-zygote args to chrome in Docker to mitigate orphan subprocess accumulation issues (also made it a lot faster as a side-effect!) 49faec8f6
[x] fix user and python binary detection config for FreeBSD 208f866, 831d82a7
[x] update automatic setup script to install via Docker/Docker-compose if possible, fall back to apt/brew/pkg/pip 06f6084e3
[x] only warn on rejected stdin instead of throwing an error 06f6084e
[x] fix bookmarklet not uriencoding urls b6ee7d291
[x] throw explicit error when django version is too low 9563e74f
[x] always try to install all dependencies on archivebox setup instead of trying to detect USE_ config options 3e7d8e52
[x] exempt /add from CSRF checking in order to support submissions from ArchiveBox browser extensions https://github.com/ArchiveBox/ArchiveBox/pull/777
[x] add support for new URL_WHITELIST option to restrict added URLs to ones matching a given regex pattern 5a2c78e https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#URL_WHITELIST
[x] #658 fix admin panel actions in safari on iOS/macOS
[x] #843 fix KeyError during some Pocket API import parses
[x] #901
[x] https://github.com/ArchiveBox/ArchiveBox/issues/971 Fix wallabag CSV parser acd53c8
[x] lots of minor UI and CLI fixes
[x] #847
[x] #22 bump Sonic version to support ARM
[x] https://github.com/ArchiveBox/readability-extractor/issues/2 bump readability-extractor version to v0.0.3 to fix relative URL rewriting
[x] #645 add option to disable original preview / all preview iframes on snapshot index pages
[x] #944 Fix broken URLs blocking entire depth=1 imports
[x] #706 fix bytes string being passed to hint causing exception
[x] #725 fix import tags being split into single characters and emptystring/whitespace-only tags
https://github.com/ArchiveBox/ArchiveBox/pull/1249
https://github.com/ArchiveBox/ArchiveBox/pull/1245
https://github.com/ArchiveBox/ArchiveBox/pull/1242
https://github.com/ArchiveBox/ArchiveBox/pull/1168
https://github.com/ArchiveBox/ArchiveBox/pull/996
https://github.com/ArchiveBox/ArchiveBox/pull/1214
https://github.com/ArchiveBox/ArchiveBox/pull/1221
https://github.com/ArchiveBox/ArchiveBox/pull/1219
https://github.com/ArchiveBox/ArchiveBox/pull/1211
https://github.com/ArchiveBox/ArchiveBox/pull/911
https://github.com/ArchiveBox/ArchiveBox/pull/1199
https://github.com/ArchiveBox/ArchiveBox/pull/1197
https://github.com/ArchiveBox/ArchiveBox/pull/1194
https://github.com/ArchiveBox/ArchiveBox/pull/1154
https://github.com/ArchiveBox/ArchiveBox/pull/1146
https://github.com/ArchiveBox/ArchiveBox/pull/1026
https://github.com/ArchiveBox/ArchiveBox/pull/1115
https://github.com/ArchiveBox/ArchiveBox/pull/1068
https://github.com/ArchiveBox/ArchiveBox/pull/866
https://github.com/ArchiveBox/ArchiveBox/pull/1021
https://github.com/ArchiveBox/ArchiveBox/pull/528
https://github.com/ArchiveBox/ArchiveBox/pull/967
https://github.com/ArchiveBox/ArchiveBox/pull/1241

TODO:

[ ] #726 fix pocket since parameter preventing failed depth=1 import from resuming
[ ] #785 fix Pinboard import not respecting original bookmarked timestamps
[ ] fix oneshot command broken: archivebox oneshot --extract=singlefile,readability https://example.com -> FileNotFoundError in log_link_archiving_finished(): [Errno 2] No such file or directory: '.../archive/1620881375.32482'
[ ] #783 fix sorting by size column in Snapshot admin causing 500 errors
[ ] #1036 fix stdin/stdout handling edge case when using nohup
[ ] #1002 fix permissions/migration creation errors

Postpone:

[ ] #696 show prompt to upgrade archivebox in UI when new version is available
[ ] #761 add support to download_url to use cookies in COOKIES_FILE (used for title fetching and other things)
[ ] #734 expose new ARCHIVEBOX_USERNAME=xyz and ARCHIVEBOX_PASSWORD=xyz env variables to make initial docker setup easier (so that manual init / manage createsuperuser isn't required as a separate setup)

Hi @pirate, It seems the Wallabag fix does not work as intended: https://github.com/ArchiveBox/ArchiveBox/issues/971#issuecomment-1122499507
Just tried it again with archivebox/archivebox:sha-03eb7e5 and can confirm this is still an issue (for me at least).

Build failed, possibly due to recent CloudFlare issues: https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc

FYI everyone watching this release, I just fixed the build, among many other things. Give it a try!

My apologies, just fixed the build and just pushed dev, (aka 0.7.0, aka latest) for amd64, arm64, and arm/v7 (run ./bin/build_docker.sh to build these yourself).

fix issue where error output was writing to >./2 instead of >&2
massively increased the speed of all the Docker builds (>20x) by adding granular shared cache mounts for reusable apt and pip caches (--mount=type=cache...)
fixed multi-arch docker builds for amd64, arm64, arm/v7 and updated Dockerfile and ./bin/build_docker.sh with latest config and PDM package manager
attempt to autodetect PUID and PGID based on the ownership of existing files within ./data, should prevent unecessary chowns and remove the need to pass PUID/PGID explicitly in many cases
and added more helpful instructions to clearly show when a user has hit an edge case that requires passing explicit PUID & PGID (like non-compliant NFS/Samba/FUSE mounts or Docker Desktop on some platforms)
updated the base image to python:3.11-bookworm-slim with bookworm-backports, node v21, yt-dlp 2023.10.13, chrome v119, and all other apt, pip, and npm dependency versions upgraded
with a lot of research, yak-shaving, and trial and error I've managed to get the Docker image down to 500~630mb for all platforms (it was >1GB in the previous version), but I cant promise it'll stay this small forever. It's a lot of work to keep it as small as possible for all architectures!
added new /VERSION.txt file containing build summary, check it on your platform for lots of juicy details about what version you're running

# pull the :latest image from docker hub (also published to ghcr.io/archivebox/archivebox)
docker pull archivebox/archivebox

# get detailed info about the build and dependency versions
docker run archivebox/archivebox cat /VERSION.txt
docker run -it archivebox/archivebox version

# run update to retry any previously failed downloads using the latest extractor versions
docker run -it -v $PWD:/data archivebox/archivebox update

hub.docker.com/layers/archivebox/archivebox/dev/images/sha256-afa301d566a01cab8934aa1141b5f6133a4585ee5b449304c659c765e240575d?context=explore

https://github.com/ArchiveBox/ArchiveBox/issues/1247#issuecomment-1788269076


Factory Code for Basic T-Display S3

What is the factory code or what do I need to change in the factory code for it to work with the basic version of the lilygo T-Display S3?

The factory code works fine with the LilyGO T-Display S3 if you edit in your router SSID and Password in the pinconfig.h!

Well, now it doesn't...
If I get it working again, I will post here!
-Terry


Port EntityEncoder[Reader] to fs2-cats

Probably a bit more advanced than #870, but still a nice opportunity for a new contributor.

Perhaps I can try this one next

👍

Fixed by #888


Fix old tests and add compatibility specs

Closes #99
Tests in test/Decapodes.jl and test/PetriNets.jl are failing due to a change in Catlab.jl behavior that the dependency CombinatorialSpaces.jl previously relied on. This will be fixed by this PR in CombinatorialSpaces.jl, but for the time being the commit 37bdb63 resolves this issue.
Compatibility specifications in the Project.toml should also be added in this branch.

Tagging a new version of Decapodes as discussed in #99 will happen outside this branch.


dispose_template calls dispose twice which can cause segfaults

Bug description
This issue was extensively discussed quite a while ago in the gtk issue tracker but never made it here. https://gitlab.gnome.org/GNOME/gtk/-/issues/5834
Within gtk-rs it can be reproduced like this:

Subclass a complex widget that calls dispose on its own children (I used adw::ActionRow but there are probably examples in gtk as well)
Use templates
Use expressions
Put the widget in a listbox
Remove the widget again

The problem is that with the current implementation it is basically impossible to opt out of dispose_template making it impossible to subclass some types with templates.
Backtrace
0x00007ffff2d58080 in gtk_object_expression_weak_ref_cb () from /usr/lib/x86_64-linux-gnu/libgtk-4.so.1
(gdb) bt
#0  0x00007ffff2d58080 in gtk_object_expression_weak_ref_cb () at /usr/lib/x86_64-linux-gnu/libgtk-4.so.1
#1  0x00007ffff7b79df7 in weak_refs_notify () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#2  0x00007ffff36f072f in g_data_set_internal () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff7b7a23d in g_object_real_dispose () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#4  0x000055555589fcb2 in glib::subclass::object::dispose<example::ui::components::task_row::imp::TaskRow> (obj=0x5555588e03f0)
    at /home/fina/.cargo/registry/src/index.crates.io-6f17d22bba15001f/glib-0.18.1/src/subclass/object.rs:160
#5  0x00007ffff7b7acc0 in g_object_unref () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#6  0x00007ffff2dbbaf5 in gtk_list_box_remove () at /usr/lib/x86_64-linux-gnu/libgtk-4.so.1
#7  0x00007ffff2dbcb86 in gtk_list_box_bound_model_changed () at /usr/lib/x86_64-linux-gnu/libgtk-4.so.1
#8  0x00007ffff7b7543a in g_closure_invoke () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#9  0x00007ffff7b8996c in signal_emit_unlocked_R.isra.0 () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#10 0x00007ffff7b8b3f1 in signal_emit_valist_unlocked () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#11 0x00007ffff7b913c1 in g_signal_emit_valist () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#12 0x00007ffff7b91483 in g_signal_emit () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#13 0x00007ffff7c9ee7d in g_list_store_splice () at /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#14 0x0000555555876cd0 in gio::auto::list_store::ListStore::splice<glib::object::Object>
    (self=0x555558626ff8, position=4, n_removals=1, additions=...)
    at /home/fina/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gio-0.18.1/src/list_store.rs:63
#15 0x0000555555876adb in gio::auto::list_store::ListStore::retain<example::ui::task_list::imp::{impl#19}::cleanup::{closure_env#0}> (self=0x555558626ff8, f=...) at /home/fina/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gio-0.18.1/src/list_store.rs:121

Calling dispose twice should be possible though, so if that segfaults isn't that a bug somewhere else?

I have always been calling dispose_template only on direct gtk::Widget subclasses. This whole function is just an unfortunate thing :(

Seems like a bug in GTK to me if it doesn't allow multiple dispose :)

Seems like a bug in GTK to me if it doesn't allow multiple dispose :)

At the moment gtk doesn't even allow single dispose, when dispose_template gets involved, since as the upstream issue explains it just …does the wrong thing
Where gtk-rs gets involved is that it's nice enough to automatically chain dispose_template - as the docs tell us to do - when using CompositeTemplate, unlike other languages which either don't support it at all or leave it for us to do
So it's kinda everyone's bug now?

Where gtk-rs gets involved is that it's nice enough to automatically chain dispose_template

We actually don't call it automatically btw.

Oh, so what is this doing then?
https://github.com/gtk-rs/gtk4-rs/blob/3807daa4c1863f299ea287aad5c95a809c34b12b/gtk4/src/subclass/widget.rs#L1416-L1431

Oh, so what is this doing then?
https://github.com/gtk-rs/gtk4-rs/blob/3807daa4c1863f299ea287aad5c95a809c34b12b/gtk4/src/subclass/widget.rs#L1416-L1431

You have to call it yourself in your own ObjectImpl::dispose override . The method exists in this trait just to move it from WidgetExt and enforce certain traits to avoid people mistakenly calling it in very wrong places

Well huh, I was certain I'd seen code that automatically did the call when the trait was implemented, guess I hallucinated that then
Mea culpa

Wasn't this resolved recently for at least the major use case: expressions?

I'm not entirely sure what you mean, but I'd be fine closing this either way. This method is a big mess, but not much we can do about it I guess.

I was talking about https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3316 but yes, I don't see much that can be done from the rust bindings side. Other bindings would have the exact same issue.


Update Get-SPEnterpriseSearchQueryDemoted.md

Update example

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sharepoint/sharepoint-ps/sharepoint-server/Get-SPEnterpriseSearchQueryDemoted.md
:white_check_mark:Succeeded
View (sharepoint-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[ADAM-2112] Refactor org.bdgenomics.adam.rdd package to org.bdgenomics.adam.ds

Fixes #2112
Replaces #2137

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3149/Build result: FAILURE[...truncated 911 B...]Wiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Using reference repository: /home/jenkins/gitcaches/adam.referenceFetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git --version # 'git version 2.25.1' > git fetch --tags --force --progress -- https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --force --progress -- https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse f008c373d85d53392c7c1adcea70b78bd99d02d6^{commit} # timeout=10Checking out Revision f008c373d85d53392c7c1adcea70b78bd99d02d6 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f f008c373d85d53392c7c1adcea70b78bd99d02d6 # timeout=10Commit message: "Refactor org.bdgenomics.adam.rdd package to org.bdgenomics.adam.ds." > git rev-list --no-walk 3075f79f6f4e02360bf5a907a3286802e66f14f9 # timeout=10Triggering ADAM-prb » 3.2.1,2.12,3.0.1,ubuntuADAM-prb » 3.2.1,2.12,3.0.1,ubuntu completed with result FAILURENotifying endpoint with url 'https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Setting status of f008c373d85d53392c7c1adcea70b78bd99d02d6 to FAILURE with url https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/3149/ and message: 'Build finished. '

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3149/Build result: FAILURE[...truncated 911 B...]Wiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Using reference repository: /home/jenkins/gitcaches/adam.referenceFetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git --version # 'git version 2.25.1' > git fetch --tags --force --progress -- https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --force --progress -- https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse f008c373d85d53392c7c1adcea70b78bd99d02d6^{commit} # timeout=10Checking out Revision f008c373d85d53392c7c1adcea70b78bd99d02d6 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f f008c373d85d53392c7c1adcea70b78bd99d02d6 # timeout=10Commit message: "Refactor org.bdgenomics.adam.rdd package to org.bdgenomics.adam.ds." > git rev-list --no-walk 3075f79f6f4e02360bf5a907a3286802e66f14f9 # timeout=10Triggering ADAM-prb » 3.2.1,2.12,3.0.1,ubuntuADAM-prb » 3.2.1,2.12,3.0.1,ubuntu completed with result FAILURENotifying endpoint with url 'https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Setting status of f008c373d85d53392c7c1adcea70b78bd99d02d6 to FAILURE with url https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/3149/ and message: 'Build finished. '

Jenkins, retest this please

Jenkins, retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3150/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3150/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3152/


Image extraction, segmentation, resampling, 2D, MRI cardiac images

Hi everyone!
Since I am about to begin a radiomic study ex novo, I just wanted to ask some questions so as to avoid annoying future problems. We will extract images and masks of interest next week from cardiac MRI with the intent of performing a future binary classification.

Should I necessarily extract images in .nrrd format or the .dcm (DICOM) format is fine and easy to convert into .nrrd?

Should the mask simply be an image with ones in the ROI and zeros outside of it?

Since we will probably work with single slices (we will select the most significant 2D slice for each patient), how should I set my third dimension since pyradiomics wants volumes as input? Moreover, which are the features that I will be able to extract? (meaning...are there some features that are only applicable in 3D?)

This is a strange question: one of the two ground truths of the final binary classification is myocarditis and this has the problem that it often doesn't appear as a single concentrated region in the image. As a matter of fact, since it is an inflammatory phenomenon, it often appears as sparse sub regions on the image. How should we deal with masks in this case? Should we just select the biggest region of inflammation? Should we somehow average the inflamed regions?

Thank you very much in advance,
Tommaso

Should I necessarily extract images in .nrrd format or the .dcm (DICOM) format is fine and easy to convert into .nrrd?

Yes, see also #361

Should the mask simply be an image with ones in the ROI and zeros outside of it?

Yes, although other values than 1 are allowed (you'd then need to specify that value in the label parameter, though). Importantly, the direction, origin and size must match to the image. If this is not the case, but the region of interest occupies a physical space contained within the image bounds, you can still use it by setting correctMask to True.

Since we will probably work with single slices (we will select the most significant 2D slice for each patient), how should I set my third dimension since pyradiomics wants volumes as input? Moreover, which are the features that I will be able to extract? (meaning...are there some features that are only applicable in 3D?)

Your third dimension is allowed to have size 1, it's just that it has to be present. All features can be extracted from 2D slices. However, take care when applying Wavelet or LoG filters, these assume true 3D input.

This is a strange question: one of the two ground truths of the final binary classification is myocarditis and this has the problem that it often doesn't appear as a single concentrated region in the image. As a matter of fact, since it is an inflammatory phenomenon, it often appears as sparse sub regions on the image. How should we deal with masks in this case? Should we just select the biggest region of inflammation? Should we somehow average the inflamed regions?

It's not a strange question. Quite the opposite, it's a very important, but difficult question. I don't have a direct answer, but you can try a large segmentation (containing all areas) or an average of multiple sub regions.

Thank you very much again @JoostJM! I'll keep in mind the mask origin/direction/size issue.
As for question 4, I will speak with my supervisor (the doctor in charge of the research project) on Monday and we'll try to figure out what makes more sense from a clinical point of view.
I was thinking that by averaging all the sub-regions we could lose the gray level dependencies, and it would also be difficult because from what I understood the sub-regions often have different dimensions and shapes. Anyway, I'll keep you updated!
Thanks again for your support!

Hi @JoostJM!
At the end, as far as myocarditis is concerned, I think we'll just select the widest inflamed region. In the meantime, since I don't have the cardiac images yet, I tried to load in PyCharm a simple brain image and corresponding mask that I downloaded from the local PACS just as a trial.
I have stored, in the same directory (All_Data_nrrd), two files which are: brain_image.nrrd and brain_label.nrrd.  Both of them have size (512,512,1) and label is of course the mask. I checked the dimension through this.
Then on PyCharm I just wrote the following code trying to emulate the "brain1" example:
from __future__ import print_function
import matplotlib.pyplot as plt
import numpy as np
import SimpleITK as sitk
import six
from radiomics import firstorder, getTestCase, glcm, glrlm, glszm, imageoperations, shape, getImageTypes, \
    getFeatureClasses, getParameterValidationFiles

imageName, maskName = getTestCase('brain', dataDirectory= "C:\\Users\\tommaso\\Desktop\\Lavoro\\Zurich_Fellowship\\Programmi\\All_Data_nrrd")

and when I run it I get the following warning:
Testcase "brain" not recognized!
What am I doing wrong? Thank you very much!

@tommydino93, that function finds the pyradiomics test cases. Try brain1.

@JoostJM I managed to run everything with the brain1 example.....but what I need now for MY brain image and for my future cardiac images will be to load images (and masks) from one of my own directories and not from the pyradiomics data test cases.
Which function could I use to do that? I thought that getTestCase could be used, just modifying the "dataDirectory" parameter..

@tommydino93, The getTestCase Function cannot be used for such a case, as it also contains code to download the test data from github when it's not found (which would be quite difficult for your data).
Easiest to use your own data is like this:
import os
datadir = "C:\\Users\\tommaso\\Desktop\\Lavoro\\Zurich_Fellowship\\Programmi\\All_Data_nrrd"
imageName = os.path.join(datadir, 'brain_image.nrrd')
maskName = os.path.join(datadir, 'brain_label.nrrd')

This sets the last two variables to the path of your data, which is similar to what getTestCase does for the pyradiomics test data (with the addition that it downloads the data if not available). It does not load or preprocess the data in any way.

Ok, thank you very much! And how about the settings (e.g. binWidth, interpolator, resampledPixelSpacing, voxelArrayShift)? Is there a rule of thumb to choose them?
P.S: would it be a problem if I commented in a fork some of the code lines? I am trying to figure it out on my own just following the code (for instance in pyradiomics/radiomics/first order) but coming from Matlab (they spoiled me too much, I admit that) I am having trouble following even the single inputs and outputs for every functions (e.s. input datatype, output datatype, parameters datatype)

@tommydino93, for that, check out the example settings in the repository. Besides that the only advice I can give you is to just see what works.

P.S: would it be a problem if I commented in a fork some of the code lines? I am trying to figure it out on my own just following the code (for instance in pyradiomics/radiomics/first order) but coming from Matlab (they spoiled me too much, I admit that) I am having trouble following even the single inputs and outputs for every functions (e.s. input datatype, output datatype, parameters datatype)

Sure no problem.

Hi @JoostJM! I managed to run this code for my brain image, following the brain1 example:
from __future__ import print_function
import matplotlib.pyplot as plt
import numpy as np
import SimpleITK as sitk
import six
from radiomics import firstorder, getTestCase, glcm, glrlm, glszm, imageoperations, shape, getImageTypes, \
    getFeatureClasses, getParameterValidationFiles
import os

datadir = "C:\\Users\\tommaso\\Desktop\\Lavoro\\Zurich_Fellowship\\Programmi\\All_Data_nrrd"
imageName = os.path.join(datadir, 'brain_image.nrrd')
maskName = os.path.join(datadir, 'brain_label.nrrd')

image = sitk.ReadImage(imageName)
mask = sitk.ReadImage(maskName)

applyLog = False
applyWavelet = False

#settings for the feature calculation. Currently resampling is disabled.
settings = {'binWidth':25, 'interpolator': sitk.sitkBSpline, 'resampledPixelSpacing':None}

if settings['interpolator'] is not None and settings['resampledPixelSpacing'] is not None:
  image, mask = imageoperations.resampleImage(image,mask,settings['resampledPixelSpacing'],settings['interpolator'])
else:
  bb,correctedMask = imageoperations.checkMask(image,mask) #bb stands for bounding box
  if correctedMask is not None: #if the mask has been correctly resampled to the input image geometry
    mask = correctedMask
  image,mask = imageoperations.cropToTumorMask(image,mask,bb)

#let's now show the first order feature calculations
firstOrderFeatures = firstorder.RadiomicsFirstOrder(image, mask, **settings) #la notazione ** serve per passare i valori del dict settings
firstOrderFeatures.enableAllFeatures()

print('Calculating first order features...')
firstOrderFeatures.calculateFeatures()
print('Done')

print('Calculated first order features: ')
for (key, val) in six.iteritems(firstOrderFeatures.featureValues):
  print('  ',key, ':', val)

Everything works fine, meaning that the program extracts all numerical first order features. What I wanted to ask you is:

Is there a way to check if the checkMask function worked properly? For instance, could I visualize (maybe with pyplot) the segmented mask?

Same thing for cropToTumorMask...how can I visualize what has been assigned to image and mask in the line image,mask = imageoperations.cropToTumorMask(image,mask,bb) and how can I be sure that it's correct?

When in the checkMask help you say "check if the label is present in the mask"...what do you mean by "label"? Do you mean the ones of the ROI? As I told you my mask is composed by ones in the ROI and zeros outside of it.

The corrected mask is usually None (That means no correction applied). The only case where it is not None is when it has to be resampled due to image geometry mismatch between image and mask (see also parameter correctMask). If the checks fail, (None, None) is returned.

image,mask = imageoperations.cropToTumorMask(image,mask,bb) returns the cropped image and mask as SimpleITK Images. You can store them as follows:

sitk.WriteImage(Image, "image.nrrd")  # Don't forget the extension!
sitk.WriteImage(Mask, "mask.nrrd", True)  # True here specifies it can apply compression

This will store it as "image.nrrd" and "mask.nrrd" in your current working directory.

Yes. Sometimes a labelmap contains multiple labels (e.g. multiple lesions). A "label" here means just that certain pixels have value 1, others value 2, etc. The check here is intended to ensure that the desired label (specified in the customization, default 1) is actually present in the mask, i.e. that there is at least 1 voxel in the mask that has the same value as label.

@tommydino93 You are missing that data because you are implementing the featureclasses directly, which is not advised. Use the featureextractor interface, it's a lot easier to use (follow the helloRadiomics.py example)

@JoostJM Alright! I've done that, thanks.
It appears that my pixel spacing is now (1.0, 1.0, 1.0) which is weird because in my original .dcm image it was different. I figured that maybe during the conversion from DICOM to nrrd something went wrong. I followed this guide for the conversion. Here are my two images opened in ImageJ:

as you can see the .nrrd image (on the right) has lost the pixel spacing (highlighted) and is now in microns and not in mm. I also unticked the "Compress" option while doing the conversion, but the problem persists.
Is there an alternative way for converting and preserving dimensions/pixel spacing?
Thanks a lot again!

You can try this, it's a script/package I built for conversion in Python. It will scan the folder and create NRRD (or NIFTII). If there is something wrong with the image, it will give you a warning what is happening.
The spacing (1, 1, 1) is usually a sign that something went wrong during conversion.

Alternatively, you can also try to see if Slicer found some issue by checking advanced in the DICOM browser and pressing the buttom Examine. After you've loaded your volume, you can also check the characteristics of that volume in the volumes model.

Thanks again @JoostJM !
I am having problems with PyCharm..maybe you can help me. I think it's a problem of configuration between PyCharm and GitHub. I typed this simple code:
import logging
import os
import pydicom
import SimpleITK as sitk
import tqdm

Source = "C:\\Users\\tommaso\\Desktop\\Lavoro\\Zurich_Fellowship\\Programmi\\Images_dcm"
Destination = "C:\\Users\\tommaso\\Desktop\\Lavoro\\Zurich_Fellowship\\Programmi\\Images_nrrd"
walk_folder(Source,Destination)

and I get as error:
Traceback (most recent call last):
  File "D:/Getting_Started_PyCharm/Pyradiomics/Prova_Conversion_to_nrrd.py", line 9, in <module>
    walk_folder(Source,Destination)
NameError: name 'walk_folder' is not defined

Did I call the function properly?
Why walk_folder is not recognized? I cloned the GitHub HTTPS in PyCharm as explained here

@tommydino93, You'll have to import the module before you can use it:
from nrrdify import walk_folder

That being said, Nrrdify also has a commandline interface. To use it, it is most easy to install it.
from the root of the package (i.e. the folder containing setup.py)
> python setup.py install
> nrrdify --help

This will print the help message detailing the usage (with optional configuration). Once it is installed, it should be usable throughout your computer. E.g.
C:\Users\tommaso\Desktop\Lavoro\Zurich_Fellowship\Programmi> nrrdify Images_dcm -o Images_nrrd

This will convert all dicoms in the folder above into nrrd and store them in Images_nrrd

Wow thanks a lot!
I managed to do it both ways. Nevertheless, the .nrrd image created is still a 512x512 microns and not a 260x260 mm as the original DICOM one :(
Could it be a problem just with the opening in ImageJ?

@tommydino93, could be. Can you share an anonymised version of your DICOMs and NRRD?

@JoostJM,  I managed to anonymize only the DICOM image (but not the .nrrd) with DicomCleaner. I hope it is sufficient for you to try it out :) Here it's the MEGA link to the image:
https://mega.nz/#!gPhHRayB
Thanks a lot again!

Sorry, that link needs a key! This already has the key:
https://mega.nz/#!gPhHRayB!JTen715bznaNqNxIzNKez_Wutb0M5RIZv1vS0_tg9BQ

As far as I can see, it appears to be caused by ITK, as it also shows the incorrect spacing of (1, 1, 1) in ITKsnap. When I looked at the tags, spacing was defined as [5.0781200E-001\5.0781200E-001], it is possible ITK is failing on the E-001, but I'm not sure. Also, I noticed the SOP class is secondaryCaptureImageStorage (usually the case for key images), do you have access to the original series? Can you try to convert that one?
As to why imageJ gives you 512 microns, I do not know. Maybe a bug of some kind? the NRRD image says the spacing is 1, 1, 1, and does not encode the Field of View size (as it can easiliy be obtained by multiplying the spacing * the matrix size.
On potential explanation is that it just does not know how to cope with the fact that there is no unit encoded in NRRD and it sort of assumes its microns (in which case 1 x 512 is indeed 512 microns).
Here are also some pointers to other tools you could try. @fedorov is currently also busy with the DICOM to NIFTII/NRRD conversion problem, this is from a project he started at NAMIC project week in Boston last January:
dcm2niix
https://github.com/rordenlab/dcm2niix
dicom2nifti
https://github.com/icometrix/dicom2nifti
dcmstack
https://github.com/moloney/dcmstack
vtk-dicom/dicomtools/dicomtonifti
https://github.com/dgobbi/vtk-dicom
FreeSurfer/mri_convert
https://surfer.nmr.mgh.harvard.edu/fswiki/mri_convert
Plastimatch/convert
http://plastimatch.org/plastimatch.html
mriconvert and mcverter
https://lcni.uoregon.edu/downloads/mriconvert/mriconvert-and-mcverter

Hi @JoostJM and @fedorov  !
Unfortunately I don't have access to the original images, because images are not exportable from IMPAX (the software they use here as PACS). I had to load the images from IMPAX to Siemens SyngoVia (a dedicated Siemens software for visualizing images) and then I extracted images from there.
Anyway, I will try the "dicom2nifti" way, hoping to solve the issue. I'll keep you updated!
Thanks again!

@fedorov , I have been trying this simple code:
import dicom2nifti
dicom_directory = "C:\\Users\\tommaso\\Desktop\\Prova_Loading"
output_folder = "C:\\Users\\tommaso\\Desktop\\Outut_Images_Nifti"

dicom2nifti.convert_directory(dicom_directory, output_folder, compression=False, reorient=False)

but no images appear in my output directory "Outut_Images_Nifti".
Could this be because all my dicom images are decompressed? @JoostJM  Could this be the problem also for the other issues I had? I mean...Do I need to extract the images as COMPRESSED rather than UNCOMPRESSED?
thanks a lot guys in advance!

@tommydino93, as far as I know, uncompressed or compressed should not matter, if anything, uncompressed should be easier. I also used this program, for the example you sent, it is able to correctly read in the DICOM and you can re-save in NIFTII (which is also accepted by PyRadiomics).

@JoostJM , @fedorov ! Thanks a lot for advising me Mango. Actually, I have great news: while doing some trials between Mango and ImageJ, I discovered that ImageJ itself allows to convert DICOM to nrrd and the pixel spacing is preserved!! We can just simply do:
File>Save As>Nrrd
And this works both for the image and for the mask. I don't know why I didn't think about it before.
Anyway, before converting all images, I just tried to load one image and one mask simply typing:
datadir = "C:\\Users\\tommaso\\Desktop\\Prova_PyCharm"
imageName = os.path.join(datadir, '87951608.nrrd')
maskName = os.path.join(datadir, 'mask.nrrd')

image = sitk.ReadImage(imageName)
mask = sitk.ReadImage(maskName)

applyLog = False
applyWavelet = False

#settings for the feature calculation. Currently resampling is disabled.
settings = {'binWidth':25, 'interpolator': sitk.sitkBSpline, 'resampledPixelSpacing':None}

if settings['interpolator'] is not None and settings['resampledPixelSpacing'] is not None:
  image, mask = imageoperations.resampleImage(image,mask,settings['resampledPixelSpacing'],settings['interpolator'])
else:
  bb,correctedMask = imageoperations.checkMask(image,mask) #bb stands for bounding box
  if correctedMask is not None: #if the mask has been correctly resampled to the input image geometry
    mask = correctedMask

  cropped_image,cropped_mask = imageoperations.cropToTumorMask(image,mask,bb)

#let's now store the image in the directory just to check whether if the cropping was successful
#sitk.WriteImage(cropped_image, "image_cropped.nrrd") #FUNZIONA, è effettivamente l'immagine croppata e quadrata

# Initialize feature extractor
extractor = featureextractor.RadiomicsFeaturesExtractor(**settings)
#enable all features
extractor.enableAllFeatures()

featureVector = extractor.execute(cropped_image, cropped_mask)

for featureName in featureVector.keys():
  print("Computed %s: %s" % (featureName, featureVector[featureName]))

and then I tried to run the shape feature program but it throws me this error:
C:\Users\tommaso\PycharmProjects\Prova\venv\Scripts\python.exe D:/Getting_Started_PyCharm/Pyradiomics/Feature_Extraction_New.py
Traceback (most recent call last):
  File "D:/Getting_Started_PyCharm/Pyradiomics/Feature_Extraction_New.py", line 14, in <module>
    image = sitk.ReadImage(imageName)
  File "C:\Users\tommaso\PycharmProjects\Prova\venv\lib\site-packages\SimpleITK\SimpleITK.py", line 8332, in ReadImage
    return _SimpleITK.ReadImage(*args)
RuntimeError: Exception thrown in SimpleITK ReadImage: C:\d\VS14-Win64-pkg\SimpleITK-build\ITK\Modules\IO\NRRD\src\itkNrrdImageIO.cxx:249:
itk::ERROR: NrrdImageIO(000001E932882740): ReadImageInformation: Error reading C:\Users\tommaso\Desktop\Prova_PyCharm\87951608.nrrd:
[nrrd] nrrdLoad: trouble reading "C:\Users\tommaso\Desktop\Prova_PyCharm\87951608.nrrd"
[nrrd] nrrdRead: trouble
[nrrd] _nrrdRead: trouble reading NRRD file
[nrrd] _nrrdFormatNRRD_read: trouble parsing space directions info |(0.89843749,0,0) (0,0.89843749,0)|
[nrrd] _nrrdReadNrrdParse_space_directions: trouble getting space vector 1 of 2
[nrrd] _nrrdSpaceVectorParse: space dimension is 2, but seem to have 3 coefficients

From what I understand, it's a spacing problem, which leads us back to my initial question: "how could I set the resampling among all my images, since they have different spacing"? E.g. should I choose the average pixel spacing among all images? Or maybe the smallest one and apply it to all other images?
Thanks a lot again, I really appreciate your help!

Moreover, must the ROI necessarily have 1 inside of it (and zeros outisde) or is it 255 also accepted?
Thanks agian

255 is accepted, but you'll need to specify it in the label parameter

Ok thanks a lot, I'll keep that in mind!
In the following mega links, the nrrd image and mask that I've extracted through ImageJ
Image: https://mega.nz/#!oLQVFYZZ!PNs-pOWlnXWM9Dau5kwrwYWnsdys4QZjka8uZxAh7ek
Mask: https://mega.nz/#!wDY0gTDI!88ZEMhDpHNkJZDfYE3o9DoRthtGi6BpxsWIMtKmU-14
In the ImageJ menu: Image>Show Info it says (like in the error that PyCharm gave me) that the space direction  is (0.89843749,0,0) (0,0.89843749,0) but I still don't understand why this is a problem for ITK
Thanks in advance!

Thanks a lot @fedorov ! You're right, I should definitely become more agile with the command line interface. It's way faster.
Anyhow, in the end I managed to convert the images from DICOM to nrrd with ImageJ, but, as I was explaining JoostJM in my previous comment, now I have a different problem with ITK which is unable to read my nrrd images, probably because of the space direction.
Any suggestion?
Thanks again for your amazing work!

Can you open the NRRD file in the text editor, and copy paste the header here?

Yes, sure. Here it is:
header.txt

I am not an NRRD expert, but it looks to me like an invalid header. Indeed, it looks inconsistent to have 3 components for the direction vectors, but only 2 vectors, and only 2 dimensions of the image.
Is your dataset volumetric, or is it a single slice? If it is supposed to be volumetric, I would really recommend you go back and try other converters. It maye well be that ImageJ NRRD writing may have issues. You could also ask ImageJ support.
If you want to just try something, you can maybe manually modify this line
space directions: (0.89843749,0,0) (0,0.89843749,0)

to this:
space directions: (0.89843749,0) (0,0.89843749)

Maybe it will fix the ITK issue ...

@tommydino93, @fedorov, I agree that the header looks weird. Additionally, changing the directions to tuples of 2 will mean that PyRadiomics does not accept it (as it only accepts volumetric sets, even if only 1 slice).
I agree with @fedorov that it is best to try to use some different tools, but if you want to manually edit the header, do so according to @fedorov's last post.

@tommydino93 if you will go back to try other converters, you might also consider converting the 3d dataset. You do not need to have just one image slice if you want to extract features from just that slice. You can just define your label in one slice, but keep the image volumetric, so that if/when you decide to look at the analysis beyond single slice, you do not need to go back and deal with the data conversion again.

Thank you both @JoostJM @fedorov !
For now I think I'll stick with the single slices just because I'm very late with the project.
I am trying to run dicom2nifti from command line and I'm having problems calling the help. Sorry guys if this is a very trivial question but I'm a newbie in command line programming. What I've done so far is:

I have unzipped from GitHub the dicom2nifti directory to D:\Getting_Started_PyCharm\dicom2nifti-master
From cmd, I've navigated to that directory with: D: and then cd Getting_Started_PyCharm\dicom2nifti-master
Once inside the directory I've typed python setup.py install
Then I've typed pip install dicom2nifti
Then I've typed dicom2nifti -h but this throws me the warning dicom2nifti is not recognized as an internal or external command

I just wanted to understand what the various [-h] [-G] [-r] [-o RESAMPLE_ORDER] [-p RESAMPLE_PADDING] [-M] [-C] [-R] mean in the line:
dicom2nifti [-h] [-G] [-r] [-o RESAMPLE_ORDER] [-p RESAMPLE_PADDING] [-M] [-C] [-R] input_directory output_directory
Thanks in advance!

HI @JoostJM ,
I think I found a work around my problem: I went back to the conversion from DICOM to nrrd with 3D slicer and I noticed, also asking in the 3D slicer forum, that 3D slicer automatically reshapes the pixel spacing to (1,1,1) for all images, which basically solves my problem. I mean....the important thing is that all images have the same pixel spacing right? In this way features extracted from them are comparable. Let me know, whenever you have time, if this reasoning makes sense :)
Furthermore, I was also trying to extract features from the LoG filtered image through the feature extractor. I tried to use this code from one of the examples:
# Initialize feature extractor
extractor = featureextractor.RadiomicsFeaturesExtractor(**settings)
#enable all features
extractor.enableAllFeatures()

featureVector = extractor.execute(cropped_image, cropped_mask)

extractor.enableAllImageTypes() #enable all possible filters (e.g. LoG, Wavelet, Logarithm, etc.)

for decompositionImage, decompositionName, inputKwargs in imageoperations.getWaveletImage(cropped_image, cropped_mask):
    waveletFirstOrderFeaturs = firstorder.RadiomicsFirstOrder(decompositionImage, cropped_mask, **inputKwargs)
    waveletFirstOrderFeaturs.enableAllFeatures()
    waveletFirstOrderFeaturs.calculateFeatures()
    print('Calculated firstorder features with wavelet ', decompositionName)
    for (key, val) in six.iteritems(waveletFirstOrderFeaturs.featureValues):
      waveletFeatureName = '%s_%s' % (str(decompositionName), key)
      print('  ', waveletFeatureName, ':', val)

but I get this error:
File "D:/Getting_Started_PyCharm/Pyradiomics/Feature_Extraction.py", line 43, in <module>
    for decompositionImage, decompositionName, inputKwargs in imageoperations.getWaveletImage(cropped_image, cropped_mask):
TypeError: getWaveletImage() takes 1 positional argument but 2 were given

Shouldn't the function take 2 parameters as input? Why it's only expecting 1 positional argument?
Thanks a lot in advance!

Hi @JoostJM , I come again. Is there a way to check if the image and mask is under the same slice? If it's possible, how can I check it? Thanks a lot!

I have another question: when I debug the code to extract features, I see the boundingBox is

but after excute the below code segment, it changed. I'm puzzled, could you tell me what has done inner the program? Look forward to your reply and thank you again.

@QiChen2014

Is there a way to check if the image and mask is under the same slice? If it's possible, how can I check it

This is done automatically by PyRadiomics. If you want to know more, check out this function (checkmask())

I debug the code to extract features, I see the boundingBox is but after excute the below code segment, it changed

The difference here is how the bounding box is defined. I use 2 separate functions for that. Internally, the bounding box is handled as a tuple specifying the lower and upper bounds. However, the bounding box that is returned as part of the provenance info is just the lower bounds as the first 3 elements. The last 3 elements there specify the size of the bounding box. E.g. for the first dimension, x, the lower bound is 213 (first element in both), the upper bound is 278 (2nd element in the internal bounding box) and the size is 278 - 213 + 1 = 66 (4th element in the provenance-returned bounding box)

I understand, thank you @JoostJM , I am very grateful!

Hi everyone! With the help of @fedorov and @JoostJM, part of the resampling problem was solved in this 3D slicer forum question.
Going back to PyRadiomics now: let's say I want a resampledPixelSpacing = [0.625, 0.625, 0]
Following the helloFeatureClass.py example, I wrote this on my PyCharm code:
settings = {
  'binWidth': 25,
  'interpolator': sitk.sitkBSpline,
  'resampledPixelSpacing': [0.625, 0.625, 0]
}

if settings['interpolator'] is not None and settings['resampledPixelSpacing'] is not None:
    image, mask = imageoperations.resampleImage(image, mask, settings['resampledPixelSpacing'], settings['interpolator'])
else:
    bb, correctedMask = imageoperations.checkMask(image, mask)  # bb stands for bounding box
    if correctedMask is not None:  # if the mask has been correctly resampled to the input image geometry
        mask = correctedMask

    cropped_image, cropped_mask = imageoperations.cropToTumorMask(image, mask, bb)

My question is:
"since the else is not computed,  checkMask and cropToTumorMask are skipped...Isn't this a problem?"

I have answered myself. By looking at the helloResampling.py it is explained that resampleImage already resamples and crops onto bounding box defined by the label!

@tommydino93 No problem!

Computationally, there is no problem. However, unless weighting by distance (weightingNorm) or force2D is used, PyRadiomics considers neighbors in-plane and out-of-plane to be comparable. If, however, z is e.g. 5 mm and xy are 1 mm, neighbours in z are 5 times further away, and therefore not exactly comparable. This then violates the comparable-neighbors assumption, upon which the feature formulas are based. There are 3 options in PyRadiomics for dealing with this:

Enable resampling to ensure isotropic voxels (see below).
Enable weighting, i.e. taking the distance into account when building the feature matrix.
Enable force2D, ignoring ofsets out-of-plane, thereby only considering neighbors that are in-plane and therefore comparable.

With resampling, you define new gridpoints in your volume, and then calculate the values at those points using interpolation (different interpolators exist, such as linear, nearest neighbor and b-spline). You can best visualize this as your volume being a continuous space, and the values of voxels being the value of discreet points in that space. Resampling means you just 'sample' the volume at different discreet locations.
This is more a philosophical question. I think I'd go for "not optimal", as there will still be valuable information extracted. It mostly reduces the rotation invariance of the extracted features.

@JoostJM Thanks a lot for the immediate answer and detailed explanation!!
So, suppose I wanted to go for option:

i. Enable resampling to ensure isotropic voxels

should I just set the three values of resampledPixelSpacing to the same number like below?
        # settings for the feature calculation.
        settings = {
          'binWidth': 25,
          'interpolator': sitk.sitkLinear,
          'resampledPixelSpacing': [0.8, 0.8, 0.8]  # in [mm]
        }

@tommydino93 Yes, that enables resampling to isotropic voxels. On what size to use, I usually advise to take a compromise with deleting information in-plane and 'creating' it out-of plane. E.g. in the case of x,y = 1mm, z = 5 mm, I usually go for 2 or 3mm isotropic. Always keep in mind that larger voxels are not necessariliy bad, as small voxels are more sensitive to noise. On the other hand, if your lesions are very small, large voxels may result in ROIs with only few number of voxels, which is also unstable.
as to your off-topic question. No gold standard. There is not even a standard (yet) on which to apply and how. Generally some are usually applied and most good radiomics packages do implement at least some.

Thank you very much again! I will open a new quick issue since it's a completely different topic.

@JoostJM
Hi,
Is there any way to convert .tiff image to .nifti. My original images are in DICOM but after creating mask using ImageJ it saved as .tiff.
Would you suggest me anyway?
Thanks

The easiest way to do it in python is to load both image and mask in SimpleITK, then set the image as the reference volume for the mask, the save the mask as niftii. One question though, is your image 2d or 3d? I believe tiff will only save in 2d, so you'd need to specify which slice it is.
Additionally, a good thing to do is to enter a vector index from the mask if it has a vector pixel type (I.e. color image). If it appears white, most likely all color channels have the value aet to 255, so the index doesn't matter.


Experiment with a raytracing approach

It might make sense to try encoding all the the fills for each tile in a texture and stepping through them in a loop in the tile fragment shader, blending each pixel in software. This is basically what an early version of WebRender did, and what @raphlinus' piet-metal does. The advantage is that we have one rendering step instead of two. The disadvantage is that we can no longer use the hardware rasterizer's fragment scheduler and we may therefore have worse thread utilization.
My hypothesis, which has a very good chance of being wrong, is that this will be slower than current Pathfinder on most high-performance desktop hardware and mobile hardware, assuming we take advantage of pixel-local storage on the latter. However, it may be faster on some lower-power desktop GPUs, such as Intel. This is likely to be very hardware-dependent.

I suspect this will not be a performance improvement. I experimented with doing the render loop in the fragment shader rather than a separate compute kernel, and performance was significantly worse. I think the main thing that's going on is that fragments from different tiles can get scheduled to the same simd group (or subgroup or wave or warp, whichever terminology you prefer), and this kills you on both thread divergence and global memory read patterns (in piet-metal, each render thread reads the same global memory location in lockstep). With a compute kernel, you get tremendous benefit from being able to schedule the work explicitly.
Incidentally, I also tried encoding "simple" tiles into the vertex->fragment data and using a point sprite (moving all global memory reads to the vertex shader), and the performance of that was also terrible. For this I don't have as much insight, as it seemed to me reasonable that the global memory reads were the expensive parts. One hypothesis is that it gets killed in register pressure, which in turn will drive down simd utilization.
All this is worth continuing to experiment with, of course.

Interesting, thanks. Really sounds like all this stuff is going to be very hardware-dependent.
I think doing the work in compute could be interesting as an alternative mode for Pathfinder on some hardware. But on mobile I would like to see pixel-local storage tried as a solution first.

Not to keep flogging the same point, but part of the appeal of compute is that it is not very hardware dependent. What actually happens during rasterization is wildly divergent from GPU to GPU, see this video for some nice visualizations. Similarly, I expect the relative performance of things like blend unit vs ALU to also vary widely. By comparison, I think it's possible to wrap one's head around the performance of compute - schedule your block sizes so you get good simd utilization and not too much register pressure, make your memory access patterns respect the memory hierarchy, avoid divergence, and you should be pretty good.
Also this issue is a good chance to point out that these systems resemble Random-access rendering of general vector graphics a bit more than MPVG - fixed size tiling rather than complex data structures, etc. I believe the RAVG work suffered from the issues of putting render in the fragment shader; to a large extent you can see piet-metal as adapting the basic ideas to compute kernels.
My gut feeling is that when this washes out, the best performance will be "if compute, then (a refined version of) piet-metal, if non-compute GPU, then PathFinder 3". It's always possible to tune things but I don't see anything yet that invalidates this idea. Of course people should also be evaluating Spinel - I've talked recently with some of the folks involved but still don't have anything remotely resembling an understanding of its performance with respect to the rest of the literature.

Well, if compute is actually faster then we should probably add it to Pathfinder as an optional path. Almost all the tiling code is equally applicable to a compute-based rasterizer, after all.
I suppose my hesitancy about compute primarily comes from the fact that it's fundamentally a bet against all the features that the hardware has specifically added for our use case, especially pixel-local storage. Pixel-local storage is precisely designed to address the main performance advantage of compute: being able to programmatically blend in fast local memory. Once you remove that advantage, you're basically betting that you can do better in software than the rasterization hardware can do, which is not a bet I'm comfortable making long-term.
One of the takeaways I've had from all of this vector graphics work is that 2D vector rasterization isn't really that different from 3D rasterization. The hardware rasterizer is essentially a 2D rasterizer to begin with.

I filed https://github.com/servo/pathfinder/issues/187 about using pixel-local storage/shader framebuffer fetch for a single pass.

We basically have this now in the D3D11 backend. Closing.


strange FirstOrDefaultAsync and SingleOrDefaultAsync results when using function in query

Project
In my project I have a separate table with UserProfile and relation to AspNetUser, sth like this:
    public class UserProfile
    {
        [Key]
        [Required]
        [ScaffoldColumn(false)]
        public string UserId { get; set; }

        public virtual ApplicationUser User { get; set; }

        [Required]
        [StringLength(80, MinimumLength = 2)]
        public string FirstName { get; set; }

        [Required]
        [StringLength(80, MinimumLength = 2)]
        public string LastName { get; set; }
    }

Problem
I often want to get just user profile so I use simple query:
var profile = await _context.UserProfile.FirstOrDefaultAsync(m => m.UserId == HttpContext.User.GetUserId()); // or SingleOrDefaultAsync

Everything works ok, I always get proper UserProfile... but when I went started to use bigger database (5k users) my code started to slow down and I've checked and fount out that this is translated to:
SELECT [m].[UserId], [m].[FirstName], [m].[LastName]
FROM [UserProfile] AS [m]

ups! there is my performance problem.
I've also found out thet when I assign userId to variable and then do the query:
string userId = HttpContext.User.GetUserId();
var part = await _context.UserProfile.FirstOrDefaultAsync(m => m.UserId == userId);

the query is ok, and looks like this:
SELECT TOP(1) [m].[UserId], [m].[FirstName], [m].[LastName]
FROM [UserProfile] AS [m]
WHERE [m].[UserId] = @__userId_0

Shouldn't EF be smart enough to fire the HttpContext.User.GetUserId() function and use it's result as parameter?

Update
I just found out that this is not only for FirstOrDefaultAsync and SingleOrDefaultAsync but many more:

Use object property as an argument.

MyObject o = new MyObject() { SomeProp = 10 };
var res = await _context.MyTable.Where(m => m.Id == o.SomeProp).ToListAsync(); // this will not even use WHERE in query. It will get all rows from table and do .Where in memory.
int val = o.SomeProp;
var res2 = await _context.MyTable.Where(m => m.Id == val).ToListAsync; // this is ok and will use WHERE.

DateTime operations

This will NOT use WHERE in query:
DateTime startDate = DateTime.Now.Date;
var res = await _context.MyTable.Where(m => m.CreateDate > startDate && m.CreateDate <= startDate.AddDays(1)).ToListAsync();

This will use WHERE in query:
DateTime startDate = DateTime.Now.Date;
DateTime endDate = startDate.AddDays(1);
var res = await _context.MyTable.Where(m => m.CreateDate > startDate && m.CreateDate <= endDate).ToListAsync();

Knowing this, I just fixed a lot of performance problems in my app


Bundler Support

Just curious if there was a way to handle gem dependency tags in ruby? I'm coming from https://github.com/szw/vim-tags which detects the Gemfile and includes the bundle show --paths automatically, and don't see a way to do that with gutentags. Is there a way to set that up natively that I'm missing, or would this be more of a feature/pull-request?
Related: https://github.com/ludovicchabant/vim-gutentags/issues/50

I think this would have to be some kind of additional mechanism that allows custom additional paths, since apparently you need to run a specific command to get the list of paths (and you probably want to have that cached until the Gemfile changes). So yeah, new feature.

That makes sense. The tricky bit would be when to run it as re-tagging the entire gemset on write would really slow things down.
For anyone else curious about working around this - I'm using gem ripper-tags which automatically generates the gem's tags, and vim-bundler to add the Gemfile's paths to vim's path.  It is working really well.

Cool, good to know, thanks.

I'll close that since I probably won't have time to write a feature I wouldn't use. Anybody's free to submit a patch with it though!


super_drag_drop not working on Android

I have downloaded your "example" app from this repo, ran the melos bootstrap command and then run the flutter super_drag_drop app. Thjis works great on Windows, with drag drop happening as expected. But on Android, the app runs but dragging and droppping does nothing at all. There is no response when I try to drop anything over the "Drop here" area, and the coloured "draggable" rectangles at the top are not draggable.
Have I missed something?

This is definitely not expected. Is there any error in the application logs?

When dragging something out got this error
[        ] I/flutter ( 4514): PlatformException(super_native_extensions_error, invalid engine handle, engineContextError, null)
[   +6 ms] I/flutter ( 4514): #0      NativeMethodChannel.invokeMethod (package:irondash_message_channel/src/method_channel.dart:45:7)
[        ] I/flutter ( 4514): <asynchronous suspension>
[        ] I/flutter ( 4514): #1      DragContextImpl.startDrag (package:super_native_extensions/src/native/drag.dart:241:9)
[        ] I/flutter ( 4514): <asynchronous suspension>
[        ] I/flutter ( 4514): #2      LongPressSession.extend (package:super_native_extensions/src/drag_interaction/long_press_session.dart:35:14)
[        ] I/flutter ( 4514): <asynchronous suspension>

When Dropping something into the app got this error
[+39729 ms] I/VRI[MainActivity]( 4514): Reporting drop result: false

Any news?

I'm also getting the same issue @knopp

E/flutter (11953): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(super_native_extensions_error, invalid engine handle, engineContextError, null)
E/flutter (11953): #0      NativeMethodChannel.invokeMethod (package:irondash_message_channel/src/method_channel.dart:45:7)
E/flutter (11953):
E/flutter (11953): #1      DragContextImpl.startDrag (package:super_native_extensions/src/native/drag.dart:241:9)
E/flutter (11953):
E/flutter (11953): #2      LongPressSession.extend (package:super_native_extensions/src/drag_interaction/long_press_session.dart:35:14)
E/flutter (11953):
E/flutter (11953):

I got this error ever since upgrading to the latest Flutter SDK (v3.24.3).
I also can't rollback anymore to a previous version, because it will cause issues on my current project.


Implement new ontologies to attach to data set metadata

From gss-cogs/gss-utils#307.
Implement new metadata to be attached at the resource level:

status
source
recommended usage
weaknesses
strengths

These new metadata fields will help attach data set-wide footnotes.

Closing this as we'll have our application profile decided on soon, we can't do anything until that's complete.


Need to Make Destination Tag to be Unsigned Long

Hello. I'm a developer and a big fan of ripple. If you will belief me, IT IS THE BEST CRYPTOCURRENCY FOR ME.
My issue is that, I'm working on developing a ripple wallet, using the official java library (xrpl4j), but I realize that most of the crypto exchange platforms uses destination tags higher than the maximum value of UnsignedInteger specified in the api.
To overcome this limitation, I hereby plead that THE DESTINATION TAG VARIABLE TYPE SHOULD BE CHANGED FROM UNSIGNEDINTEGER TO UNSIGNEDLONG.
Thanks in anticipation of a relieving response.
Love you all.

Hi @txysystems, I'm not aware of any exchanges that use destination tags larger than a 32-bit unsigned integer. In fact, I'm not sure how that would be possible, considering the XRPL protocol enforces that destination and source tags are 32-bit unsigned integers. the UnsignedInteger class represents a 32-bit unsigned integer, so should be large enough to hold any destination tag.
Can you tell me which on which exchange specifically you are seeing these larger destination tags?

Closing this as wont-fix since UnsignedInteger holds unsigned integers up to 2^32-1 (i.e., that can fit into 32-bits), and per the docs here:

In transactions, both source and destination tags are formatted as 32-bit unsigned integers.

You can also see in transactions (e.g., a Payment) that destination tags are typed as UInt32.
Please re-open or make a comment here if there's still a concern.


Fix population graphs

Description
Steps to reproduce
Note the bug depends on the state of the browser (probably cookies, although I have not investigated this further)

Visit http://www.ensembl.org/Homo_sapiens/Variation/Population?db=core;r=1:230709548-230710548;v=rs699;vdb=variation;vf=179
Clear site data for ensembl.org (in Chrome: open dev tools > Application > Clear site data)
Refresh the page (make sure that there is no hash value in the end of the url)
You should see population charts:

Open one of the sub-populations
Refresh the page again
The charts will likely stop rendering

Cause of the bug
If the browser does not have certain values preserved in its state (🤷♂) then loading the page will trigger the toggleable function in Content.js, which will simulate click event on any elements that have been "opened", such as the subpopulation on this page. This will fire the toggleContent method on PopulationGraph, which will call the makeGraphs function. The problem here is that for the makeGraphs function to work, the getContent method must first have executed; otherwise there won't be necessary data in the Piechart class, and makeGraphs will fail.
Solution
Check whether the getContent method has completed (by an attribute that it sets on the Piechart class) before running makeGraphs. It may not be a very pretty solution, but since this code will eventually be replaced, it's probably ok.
Another change
Remove the hash anchor from the url when a toggleable element is turned off.
Views affected
Population genetics for variant.
View in sandbox: http://ves-hx2-76.ebi.ac.uk:8410/Homo_sapiens/Variation/Population?db=core;r=1:230709548-230710548;v=rs699;vdb=variation;vf=179
Possible complications
🤷♂
Related JIRA Issues (EBI developers only)
https://www.ebi.ac.uk/panda/jira/browse/ENSWEB-5503

@azangru
By removing the hash from the URL that means when you refresh the page you wont be able to go that section again?
I am not sure i understand your solution. what about using the defer function to execute the makegraph after getcontent has been done?

By removing the hash from the URL that means when you refresh the page you wont be able to go that section again?

Yes. But the hash should get removed when a toggleable element is "closed". That should signal the user's intent not to view this section anymore.

I am not sure i understand your solution. what about using the defer function to execute the makegraph after getcontent has been done?

I am preventing the PopulationGraph module from executing toggleContent on initial page load entirely. It seems that there is no need for it to do that anyway. The way this works is that PopulationGraph checks for the field (this.graphEls) that gets created during execution of getContent. If there is no this.graphEls field, this.makeGraphs should not be executed.


Add text to icon buttons

I accidentally published a post because I thought the icon meant “Save to GitHub”.

Before:

After:

I like the idea of Textual labels, but perhaps keep them as hover-text, or long tap in Android.
If you were using this on a Tablet in Portrait, those text labels would be cut off. Particularly with release 0.5.1 now having an extra icon in the blog editing view.
I've noticed that if you long tap on the icons in Android Marshmallow, Google Now On Tap displays the original file name of the icon pack, not the actual function of the button.
@anthonny should fix that behaviour so it meets Accessibility guidelines.

I like this idea too.
There a big rework to do with the UI, in the first the UI was handcrafted and in the current version i use material-ui.
But for me it is not beautiful, there is not any identity instead of the logo and the orange color :)

If you make people hover despite this much empty space in the bar (on desktop), people might get annoyed; there is responsive design (CSS media queries) to show/hide element parts based on available space 😋
Another ambiguity:

there is responsive design (CSS media queries)

Yes, i use MQ to adapt the view on mobile only. Like i said i did not invest a lot of time on the design of HubPress 😄
If you have some skills in UX/UI and have time, feel free to make some proposal 👍
I agree with you the pen is not the best icon to say "new post", a plus should be better :D

@anthonny Where could I check out the code that generates those UI buttons? The app.js file is a minified bundle 😅

Everything is here https://github.com/HubPress/dev.hubpress.io/tree/development
It's a React app using Redux and Material-ui

Feel free to join our slack http://hubpressio-slack.herokuapp.com if you have any question

It is now a vue2 app. 😁
I still think this is a valid issue. The textual lables under each icon would be better in mobile browsers because you can't hover over the button.

I agree with you @jaredmorgs it is still a valid issue


Object.isFunction is broken for iterators (and, I'm sure, something else)

As I see, Object.is<type> is something like this.
function isType(el, type) {
  return Object.prototype.toString.call(el) === '[object ' + type + ']';
}

It's really really bad)
Object.prototype.toString(Set.prototype[Symbol.iterator]); // "[object Object]"
typeof Set.prototype[Symbol.iterator]; // "function"

P. S.
And guys, could you DO NOT rewrite native functions if they exists? Do not rewrite at all.

+1

And guys, could you DO NOT rewrite native functions if they exists? Do not rewrite at all.

They have different semantics, so they should be overriden

They have different semantics, so they should be overriden

I told about the methods that marked as polyfills in the docs (Array.isArray,Array#filter, Array#map, etc.). Their semantics is totally equals to the native implementations.

It's really really bad)

You're going to have to provide evidence that it's "really really bad", not just assertions. So far, Object.prototype.toString is a reliable way to do native checking and avoids a number of pitfalls that can be found with both the typeof and instanceof operators (these are well documented). Many performance enhancements have been tried and abandoned due to issues with these operators, and right now it's the most reliable type checking method we have (one notable exception here is Arguments objects in older IE, a number of hacks are in place in various libs to do type checking on arguments that are approaching reliable, but there is no 100%).
Also, there is a problem with your example. You are calling toString without passing context, which will always result in [object Object]. The point is interesting, however, and I think it's important to keep our eyes on the spec regarding both built-in iterators and generators. If at any point they become something other than "functions", it may be necessary to call into question whether isFunction needs to distinguish between normal functions and generators. For now, however, I have tested in all major ES6 enabled browsers and they are correctly returning [object Function].

DO NOT rewrite native functions...(Array.isArray, Array#filter, Array#map, etc.)

Part of your complaint is valid, but "rewriting" is an oversimplification. First, lets set aside Array.isArray, as this method is a true polyfill, and will always use browser implementations if they exist.
In contrast, Array#filter, Array#map, etc are "enhanced" to allow a first argument other than functions. This allows for shortcuts to mapping functions (for map) or matching functions (for the rest). These enhancements are only being applied if the first argument is not a function, in which case the native implementation would throw an error. If the first argument is a function, then it will hand off to the native implementation (or polyfill if none exists). In fact, the enhancements simply build functions that can be handed to the native implementation so every time any of these methods are called, they will be in the end handled by the native implementation.
The fact is that these enhancements are one of the most powerful and expressive parts of Sugar, and were not added on a whim. That being said, I can understand the controversy behind doing anything beyond polyfills for native methods, and I think that the complaint here is valid.
The next version of Sugar (2.0.0) will be a major rewrite. The biggest change will be that NO prototypes will touched by default -- this feature will become opt-in.  Sugar also does not touch Object.prototype by default due the major problems there, so this will become an option to the "extend" method. My proposal is that these array enhancements also become their own option. It should look like this (option names are not final yet):
Sugar.extend({
  objectPrototype: true,
  arrayEnhancements: false
});

This will allow fine-grained control over some of the more "controversial" parts of Sugar and should address your problem nicely. Object.prototype enhancement will be off by default. Whether or not array enhancing is as well is something I will have to think about and will depend on what people are expecting, etc. The best part, however, is that this way the enhanced methods will still be available through the global object, like in Underscore, etc:
Sugar.Array.filter([1,2,3], 1);

Also I should mention the above ONLY applies to every ,some, filter, find, findIndex, and map on arrays. Every other Sugar polyfill not only uses native implementations but also is thoroughly tested and should have identical behavior (including the polyfills for the above methods which exist separately from the enhancements).

Also, there is a problem with your example. You are calling toString without passing context, which will always result in [object Object].

It's just mistype, yes, but... Shit, I don't remember, how did I get It with the right example. I'll try to create working example again :(

The next version of Sugar (2.0.0) will be a major rewrite. The biggest change will be that NO prototypes will touched by default -- this feature will become opt-in.

Awesome, thank you, guy) Sorry for my irritable behavior when I wrote this issue. This bug knocked me out)

I don't remember, how did I get It with the right example

Definitely let me know. It raises a good question. I wouldn't be surprised if things still need time to settle around iterators/generators, but if there's spec behavior that makes this tricky as well then it should be handled.
Also if there really are issues around Object.prototype.toString for other types, I am open.

Closing this for now. Note that is methods for primitive types isString, isBoolean, isNumber are returning quickly up front on a typeof to help with perf. For the record I'm not doing this with functions as I am not quite ready to part with support for early Firefox versions that incorrectly report regexes as type "function".


Form field text values too large

Link to PDF file (or attach file here): detect-form-fields.pdf
Configuration:

Web browser and its version: Firefox 46.0.1
Operating system and its version: Ubuntu 16.04
PDF.js version: 1.3.231
Is an extension: No

Steps to reproduce the problem:

Open PDF
Look at entered text in form sections 8a and 9a

What is the expected behavior? Text is correct size:

What went wrong? Text is very large:

PdfBug console output:
11:56:13.291 PDF ef2293a3b9f4e0accbef97ee38798a2c [1.3 Mac OS X 10.11.1 Quartz PDFContext / Amgraf OneForm Designer Plus] (PDF.js: 1.3.231)1 viewer.js:6800:7
11:56:13.292 Warning: AcroForm/XFA is not supported1 viewer.js:6824:9
11:56:13.424 Warning: Unimplemented annotation type "FreeText", falling back to base annotation1 pdf.worker.js:9407:5
11:56:13.806 Warning: fontRes not available1 pdf.worker.js:9407:5 (logged 54 times)
11:56:13.898 Warning: Error during font loading: Font Helvetica-Bold is not available1 pdf.js:368:5

I have the same issue right now. I am inserting values in the proper place, and it automatically uses bigger font than I am using. However after downloading and saving the pdf using pdf js, the issue dissapears when using acrobat reader or foxit.

I have a similiar problem, but I do not yet know what to do about it:
If this is a problem with AcroForms support within PDF.js, why is this ticket (https://github.com/mozilla/pdf.js/projects/1) resolved/closed while this one here is still open?
I can still reproduce this in version 2.3.171.
Is there a chance to circumvent the problem somehow, i.e. by installing fonts etc.?

The problem with the file here is that the appearance streams are missing. /NeedAppearances is set so the viewer should generate them. In the field "f1_13(0)" the /DA (default appearance) is "/Helvetica-Bold 9 Tf 0 g". However the font "Helvetica-Bold" does not exist in in the Acroform default resources of the PDF (I suspect that the PDF was poorly merged from a cover page and the original SS4 form). So PDF.js makes some decision about what font to use, and from the screeenshot this looks like Comic Sans 🤣
@hveit make sure that the software that generates / fills the PDF also creates the appearance streams of the fields. If you're not sure what is going on, create a new issue here.
@timvandermeij you closed this issue but no explanation why.

@THausherr thanks for the explanation.
Can you confirm, that it is the same problem with my PDF file from issue #11151?

@hveit I'll answer there.

This looks much better now, but the values are cut off a little:


Log when a path can't be looked up from fsid

Remove the crash
Add logging around this case

resolves #959

@nickgra I would greatly appreciate it if you can pull/build/test this branch against your repro.  It will also give us insight into which messageType is failing.

Also, are we still maintaining the mirror provider? If so, I think the change to _PrjFS_Callbacks will break that; the mirror provider will need to implement the OnLogError callback as well.

I could use more info on the MirrorProvider impact here.  I would expect it to run, but not do anything for this event.  Considering we're doing this as a patch and it may be a short lived function, I would think it's OK to skip MirrorProvider for now.  Am I missing anything?  @wilbaker

I could use more info on the MirrorProvider impact here. I would expect it to run, but not do anything for this event. Considering we're doing this as a patch and it may be a short lived function, I would think it's OK to skip MirrorProvider for now. Am I missing anything? @wilbaker

@pmj is correct that we will break MirrorProvider with this change if we don't update it.
Since the update to MirrorProvider is very minor (it just needs to implement OnLogError and register it as a callback) I think we should update it as well.

I could use more info on the MirrorProvider impact here. I would expect it to run, but not do anything for this event.

Well, PrjFS_StartVirtualizationInstance() explicitly fails if OnLogError is null. If you want it to fall back to doing nothing, you'll need to remove that check and instead guard for null when calling the callback.

Considering we're doing this as a patch and it may be a short lived function, I would think it's OK to skip MirrorProvider for now. Am I missing anything? @wilbaker

Yeah, I can't comment on the status of where this patch is ending up, all I'm seeing is that this PR is attempting to merge into master, so I assumed it was going to stick around.

FWIW, latest round of changes looks good to me.


[WOR-149] Add cloud context icon

https://broadworkbench.atlassian.net/browse/WOR-149

<!--
Hello, friend!
Remember to mention what sort of testing/verification you performed in the course of building this PR, i.e. checking functionality in the browser locally.
Also, so if a screen recording [1] and/or screenshots would be a helpful way to communicate context, please consider including some in your PR description.
Thanks!

[1] https://support.apple.com/guide/quicktime-player/record-your-screen-qtp97b08e666/mac
--!>

@yunip20 can you also change the "recently viewed workspace card" to use the more concise icons?

Nit, do we want icon sizes to be consistent between the cards and the list view? FYI @MatthewBemis


MM-18791: Add spacing to Helper text on Settings Screen

Added marginHorizontal spacing to the footer for sections.
Summary
Added a marginHorizontal to the footer text found in the Section element to resolve the spacing issue on the helper text.
Ticket Link
https://mattermost.atlassian.net/browse/MM-18791
Checklist

[x] Has UI changes - 15px horizontal margin to footer for section element

Device Information
This PR was tested on:
iPhone X 12.4 (Simulator)
iPhone Xs Max 12.3
Screenshots

/cherry-pick release-1.25


Error when compiling css file

Hi there,
(hoping not to open an issue for a simple error of me)
I'm getting an error when trying to compile a css file.
I am running npm run hot
on a js file containing import '../../../theme/index.css'
ERROR in ./theme/index.css
Module parse failed: /Users/rd/Documents/Tchaka/Development/TchakaCM/code/web/theme/index.css Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
| .el-form-item__content:before, .el-form-item__content:after {
|     display: table;
|     content: "";
 @ ./resources/assets/js/app.js 5:0-34
 @ multi (webpack)-dev-server/client?http://localhost:8080 webpack/hot/dev-server ./resources/assets/js/app.js

I think it tries to interpret the css file as javascript.
You can simply use mix.combine() or mix.sass() for your styles.
If you want to be able to import css in your javascript, you should add a new rule for that in your webpack.config.js.

Yeah, right now, we don't have a rule for importing CSS into your JS files. I'm not against adding it, though, if a lot of people want to do it.
Until then, you can manually add it in, like this:
mix.webpackConfig({
        module: {
            rules: [
                {
                    test: /\.css$/,
                    loaders: ['style-loader', 'css-loader']
                }
            ]
        }
   });

If lots of you want this added by default, "like" this reply.

Thanks !


Monochrome Quake1

Hello @Kivutar or somebody who made this set, can you look into this file?
Quake1.svg does not have same width and height.
Quake1.png may come from Quake1.svg but does not seems consistent.

Monochrome does have icons for New! Pokemon Mini.

Hi, please let me know if my pull request fix the issues.
About Pokemon Mini... can you confirm that "Pokemon mini" is this gadget (I don't know it)?
https://it.wikipedia.org/wiki/Pokémon_mini#/media/File:PokemonMini.JPG
I'm totally busy, but I could try to create the icons.

Yes on svg+png.
Yes on Pokemon Mini.
A tiny gadget with 96 x 64 pixel monochrome LCD. There are about 10~ games total too (not counting regions). Pokemon Mini + Content: http://mlm-d2-p.mlstatic.com/game-boy-pokemon-mini-de-nintendo-en-excelente-estado-14849-MLM20090533155_052014-F.jpg?square=false

Thank you for fixing this (just saying).
I also made horrible temporary Pokemon stuffs.
https://github.com/lasers/retroarch-assets/blob/5dc22d0b0a1ef36d9e6eba511505a65c8398410c/xmb/monochrome-jagged/png/Nintendo - Pokemon Mini.png
https://github.com/lasers/retroarch-assets/blob/5dc22d0b0a1ef36d9e6eba511505a65c8398410c/xmb/monochrome-jagged/png/Nintendo - Pokemon Mini-content.png

Have you used raster or vector graphics?
Could you share the svg file? You are on the right way (if you used a vector graphics software like Inkscape). Just a few improvements and your "stuff" could be ok. ;)

I don't know who made Monochrome. Is it you? I made/prefer a jagged version of Monochrome with a command so it should stay jagged. I don't have svg, just xcf. If you look closely, it's not that accurate. I made them with Gimp. https://transfer.sh/3ezMq/npm-gimp-7z
Stock Photo: http://pocketmedia.ign.com/media/hardware/image/pokemonmini/picture_2.jpg

I just added the Pokemon Mini stuff in my previous Pull request. Please give it a try and let me know if all is fine.
I think that the original guy who created Monochrome icons is bibalox. I'm just a contributor.
For this job you should use a vector graphics software (like Inkscape) instead of a raster ones (Gimp).
If you do this you can create an SVG file (so you can zoom it without quality lose) and export a .png file from there. In this way you can leave also a source file (.svg) to the Retroarch community for further mod or improvements.

Giving it a try! Cute! I don't know If I should be accurate? I'll list what I see.
Pokemon Mini:
D-pad might be a little too thick/square. The power (top middle) button might be a little too big when compared with the buttons on right side. The reset (bottom middle) button looks great! Maybe enlarge it just a teeny bit so it does not look like a tiny dot on the screen...
Pokemono Mini Content:
The Pokemon content looks off comparing to games.jpg. I think it might be better if the sides were straight to make it less mushroom... And if it's possible, implement thin line (curvy) line somewhat like my content.xcf to give it some identifying characters... along with a nip on right side (Something I didn't notice before).
I'm not familiar with Inkscape at all... but I can try next time.

Check it now. Better than this is not possible.
:D

Two thumbs up signaling "This is too perfect!"  👍 👍 😉

Thanks. I added jagged versions just now.

Perfect. See you ;)


chore(ci): add E2E test workflow

Adding workflow that runs e2e test suite (#76) after push to merge event and on demand. It does not replace PR check that will be added as well.

@cbr7 If you would please take a look.

Fixes #107


Gabe optional ipmi power restrictions

@Primer42 @maddalab @schallert @roymarantz @tumblr/systems
This allows collins to perform power commands on assets that do not have IPMI Information configured, if you set powermanagement.allowAssetsWithoutIpmi = true.

Looks like the tests failed. I reran them thru Travis and they passed :/ Ill update this branch to trigger a new build.

There is a flaky test mostly around solr index updates when on travis. Takes some time for indexes to update causing some tests to fail (some times)

LGTM

:+1:

This looks fine in itself, however I see an exception being thrown please test what happens in the gui for Power Status and Identify on exception to ensure a proper message is displayed

@maddalab I have tested and it works as expected (the exception percolates up to the UI, just like if there is a timeout.)


Don't include MiqRequestMixin in the RSpec example group

This seems to have been added as a part of f551071ff62.
@syncrou Can you take a look?


Notifications not marked as read on clicking

When new notifications are created, clicking on them from the notification bubble will not mark them as read, instead the user has to click on the "View All" option.
Steps to reproduce:

Go to http://dev.kryptedgaming.com/games/
Next to Eve Online, click the "+" and "x" symbols several times
New notifications will be created for the user adding and removing a game to/from their account.
Click on the new notification icon at the top left of the screen, which will expand the list of unread notifications.
Click on any notification in the list.
Notice that the number of unread notifications remains the same and the list does not become smaller.

Will need to make an AJAX call on the button press of notifications.


Not able to use Configuration file for connecting to mongo-connector

Team,
Not sure this is the right place for asking questions. Please excuse me if its not correct place.
I'm trying to establish a connection between 2 mongo replica sets through mongo connector and its working fine. But instead of mentioning the server name and port details in cmd line argument, I want to configure it in a config file.
mongo-connector -m localhost:27017 -t localhost:37017 -d mongo_doc_manager
I followed the steps mentioned in the link [](](https://github.com/mongodb-labs/mongo-connector/wiki/Configuration-Options)  but its not working. I'm also pasting my config.json file here. Please correct if i'm making any mistakes.
I want to pass command like this mongo-connector -c myconfig.json instead of mongo-connector -m localhost:27017 -t localhost:37017 -d mongo_doc_manager
myconfig.json file
{
    "__comment__": "Configuration options starting with '__' are disabled",
    "__comment__": "To enable them, remove the preceding '__'",

    "mainAddress": "localhost:27017",
    "oplogFile": "C:\Dev\mongodb\mongo-connector\oplog.timestamp",
	"noDump": false,
	"batchSize": -1,
    "verbosity": 2,
    "continueOnError": false,

    "logging": {
        "type": "file",
        "filename": "C:\Dev\mongodb\mongo-connector\mongo-connector.log",
		"__format": "%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s",
        "__rotationWhen": "D",
        "__rotationInterval": 1,
        "__rotationBackups": 10,
        "__type": "syslog",
		"__host": "localhost:27017"
    },

    "docManagers": [
        {
            "docManager": "mongo_doc_manager",
            "targetURL": "localhost:37010",
            "__autoCommitInterval": null
        }
    ]
}

The only thing that stands out to me is your targetURL port number is 37010 instead of the 37017 in your examples. Are there any errors reported in mongo-connector.log?

Thanks for the response. I have corrected it in myconfig.json but still getting same exception.Please find below the exception details.
C:\Dev\mongodb\mongo-connector>mongo-connector -c myconfig.json
Fatal Exception
Traceback (most recent call last):
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 110, in parse_args
self.load_json(f.read())
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 132, in load_json
parsed_config = json.loads(text)
File "C:\Program Files\Python\lib\json_init_.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Python\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Python\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 6 column 21 (char 201)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\util.py", line 90, in wrapped
func(*args, **kwargs)
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\connector.py", line 1059, in main
conf.parse_args()
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 112, in parse_args
reraise(errors.InvalidConfiguration, *sys.exc_info()[1:])
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\compat.py", line 9, in reraise
raise exctype(str(value)).with_traceback(trace)
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 110, in parse_args
self.load_json(f.read())
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 132, in load_json
parsed_config = json.loads(text)
File "C:\Program Files\Python\lib\json_init_.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Python\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=w(s, 0).end())
File "C:\Program Files\Python\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
mongo_connector.errors.InvalidConfiguration: Invalid \escape: line 6 column 21 (char 201)
Traceback (most recent call last):
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 110, in parse_args
self.load_json(f.read())
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 132, in load_json
parsed_config = json.loads(text)
File "C:\Program Files\Python\lib\json_init.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Python\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Python\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 6 column 21 (char 201)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python\Scripts\mongo-connector-script.py", line 9, in
load_entry_point('mongo-connector==2.5.0.dev0', 'console_scripts', 'mongo-connector')()
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\util.py", line 90, in wrapped
func(*args, **kwargs)
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\connector.py", line 1059, in main
conf.parse_args()
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 112, in parse_args
reraise(errors.InvalidConfiguration, *sys.exc_info()[1:])
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\compat.py", line 9, in reraise
raise exctype(str(value)).with_traceback(trace)
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 110, in parse_args
self.load_json(f.read())
File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 132, in load_json
parsed_config = json.loads(text)
File "C:\Program Files\Python\lib\json_init_.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Python\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Python\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
mongo_connector.errors.InvalidConfiguration: Invalid \escape: line 6 column 21 (char 201)

As you aware, I'm trying this in windows environment.

You need to escape your backslashes, eg.
"C:\\Dev\\mongodb\\mongo-connector\\oplog.timestamp"

Hurray!!! It worked this time. Silly mistake killed my time. Thank you very much for resolving this.


Fix environment variables setup for langflow

This PR provides some documentation update:

Removes the old LANGFLOW_ prefix of the environment variables in the Langflow setup instructions.

[!IMPORTANT]
Removes outdated LANGFLOW_ prefix from environment variables in Langflow setup documentation.

Documentation Update:

Removes LANGFLOW_ prefix from environment variables in langflow.mdx setup instructions.
Updates examples to use LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST without the prefix.

This description was created by  for 4ff83ee26920c79e26c2c9743ba64a3832afc31d. It will automatically update as commits are pushed.

@sgobotta thanks for raising the PR. Did you try whether this works end-to-end?


Reduce cost of in memory iptables management

Fix management of in memory copy of iptables. Only manage ipv6 iptables if ipv6 enabled and use rule tagging to remove rules instead of doing a search of rule definitions. For a compute host with 450 ports these changes dropped processing time for deconstruction of iptables by 81% and overall iptables reconstruction by 70%.

Tests completed:

tox -e pep8
tox -e py27
performance tests

Defect for pull request is: https://github.ibm.com/alchemy-containers/signature-experience/issues/30


"Esc" key doesn't hide the completion widget on the IPython console

From ccordoba12 on 2013-08-20T10:41:21Z
The title says it all. I've spent several days trying to find why the widget doesn't receive Esc or where it's get lost, but without luck.
Pierre, could you help me with this one?
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1524

From ccordoba12 on 2013-08-20T08:43:15Z
I should mention that this works on a regular qtconsole and also worked in our 2.1 qtconsole integration.

From ccordoba12 on 2013-12-18T06:52:43Z
This issue was updated by revision 798af4fc94f1 .

Before "Esc" was registered as a QShortcut of its parent widget and hence it
could be easily hidden even if it was unfocused.
But this was clobbering "Esc" key actions of the parent. For example, the
IPython console has several of them that were not working correctly.

Status: Fixed


Auto uninstall apps

Brevent is auto uninstalling apps...like last night it uninstalled 3apps of mine...how frustrating...I will lose those data forever....Also brevent was disabled from settings all this time...

Brevent never auto uninstall any apps.


chore: Adding semantic release

Blocked on https://github.com/go-semantic-release/action/pull/7

Forgot semantic-release does not support mono repos


fix: don't require optional peerDependencies

@vue/server-renderer and @vue/compiler-dom were both required in the
sense that they will throw regardless of whether or not your are using
functionality from those libraries if they weren't installed. I don't
believe this was intentional, so I have made them optional and they will
now only throw if you try to use them and the package isn't available,
and will print a more helpful error message when doing so.

How can I reproduce and test the issue this supposedly fixes? I am not comfortable merging this as it - I don't know what it fixes or how to test it.

I'll see if I can create a small reproducer.

I created a test repo that demonstrates 3 separate issues across yarn, pnpm and npm that are related to the issue in this PR. I was really hoping to just see the same problem manifest itself in the same way 3 times, but funny enough each package manager seems broken in slightly different ways.
https://github.com/thebanjomatic/test-utils-repro/

Today (after updating test-utils) I ran into the issue fixed by this PR.
pnpm has stricter package isolation than classic npm or yarn, so when unconditionally importing the module without having it as a dependency, it will always fail.
For now my workaround is to add
{
 …
 "pnpm": {
    "packageExtensions": {
      "@vue/test-utils": {
        "dependencies": {
          "@vue/server-renderer": "3.4.15"
        }
      }
    }
  }
  …
}

to my package.json.

@thebanjomatic, do you believe that your pull request still necessary?
@lmiller1990, @cexbrayat, could you say if the issue this pull request is trying to solve still valid?
If the answer is not, I think we should close it to keep the repository clean with only real problems.

I am not sure. I am not deep enough in the package management ecosystem to know, and I have never had an issue using Vue / Test Utils with any of the projects I'm working on.
Since it looks like there isn't a pressing need for this and no-one else has raised issues, let's just close it for now.


Writing over LRC files in Jellyfin

Full disclosure, this might just be something I am doing wrong.
I was thinking if you added a "If it doesn't exist" piece of code to these lines it should fix it.
"""
if isAudioFileExtension(file_extension) and lrc_for_audio_files:
write_lrc(result, file_name + '.lrc')
"""
Into something like below
"""
if isAudioFileExtension(file_extension) and lrc_for_audio_files:
if exists(): # the file already exists
pass
else:
write_lrc(result, file_name + '.lrc')
"""

To be clear, it's overwriting the LRC files every time?  If so, I might be missing a check, similar to one I have for SRTs.

If my above statement is true, I may have fixed it.
https://github.com/McCloudS/subgen/commit/5b5fba3d82f047cbf3fd216dea92680a3d511bdb

This solution works. If I wanted to get subtitles in Spanish instead of English, how would I do that?

Oh that's a bummer, thanks for letting me know.